Core VIBE Engine Architecture¶

Architectural reference for the core VIBE document assembly engine. Optimized for LLM consumption.

See also:

• components.md - Component system details
• assistant.md - AI-assisted interview system
• frontend.md - Frontend UI patterns (htmx, Alpine.js)
• llm-providers.md - LLM provider abstraction
• review.md - Document compliance review
• author.md - Template authoring workspace

1. SYSTEM OVERVIEW¶

VIBE is a Flask-based document assembly system that generates documents through dynamic interviews. Core innovation: probing mechanism determines question relevance by executing templates with placeholder values rather than static analysis.

Core Philosophy: Template-as-Truth

• Templates control question relevance (not configuration)
• Execution-based discovery (not prediction)
• Server-driven UI with minimal client logic

Application Core: VibeFlask (subclass of Flask in vibe/core.py) adds type-hinted attributes for template_provider, LLM_ENDPOINTS, etc. Infrastructure uses DataDirectoryManager (vibe/infrastructure/data_directory.py) for all data path resolution (sessions, logs, caches). Config path is resolved from VIBE_CONFIG_PATH env var (default: config.yml), overridable via vibe run --config path/to/config.yml.

2. PROBING MECHANISM (CORE INNOVATION)¶

2.1 Central Architecture¶

The probing system determines question relevance by executing the template with current state and tracking variable access via custom Jinja Undefined handler.

Key Files:

• vibe/probing/orchestrator.py - Core probing logic and TrackingUndefined
• vibe/web/core/state_tracker.py - Flask wrapper and relevance determination
• vibe/probing/exceptions.py - ProbeMissingPathError, ProbeNeedsError

2.2 Core Function: run_probe¶

Location: vibe/probing/orchestrator.py::run_probe (delegates to ProbeOrchestrator). Accepts optional alias parameter for component probing.

Algorithm (via ProbeOrchestrator):

1. Configure TrackingUndefined as Jinja undefined handler
2. Initialize probe_context (NestedValue with tracker callback)
3. Set up PlaceholderManager for on-demand placeholder creation
4. Install _ProbeVibeContext (closure-based VibeContext subclass) as env.context_class. Internal keys (_host_template, _is_probing, _probe_runtime, etc.) come from the closure, not from the data dict.
5. Suppress NestedValue tracking -- Before template.render(probe_context), set suppress_nv_tracking contextvars flag (vibe/nested_value.py) to True. Jinja's Template.render() internally calls dict(probe_context), which invokes __getitem__ on ALL top-level keys -- this would create false-positive accesses. The flag is reset in _ProbeVibeContext.__init__ (which fires after dict() but before actual rendering begins).
6. Render template -- _ProbeVibeContext is constructed automatically by Template.render() with probe_context as the parent dict:
   • TrackingUndefined instances resolve placeholders on first use (no immediate retry needed)
   • _ProbeVibeContext.resolve_or_missing() catches top-level variable accesses that bypass NestedValue dunder methods (e.g., Jinja tests like is true, is none, passing values to filters/macros) and records them via the orchestrator's tracker
   • Only retry on ProbeNeedsError (component needs additional context)
   • Only retry on ProbeMissingPathError for local variables (component aliases, appendix paths)
7. Restore original Jinja environment (undefined handler and context_class)
8. Return ProbeResult

TrackingUndefined Pattern:

• Custom Jinja Undefined class installed as env.undefined
• On undefined variable access, TrackingUndefined is instantiated
• First operation (str(), bool(), etc.) triggers _resolve_value() via PlaceholderManager
• Records access path + template location (file, line number)
• Delegates all operations to resolved placeholder value
• Child attribute/item access spawns new TrackingUndefined with extended path

Important Probe Semantics: Probing is short-circuit -- it tracks what was executed for the current state, not all possible branches. accessed_keys means "touched during this probe run". Comparison hooks run on both unresolved (TrackingUndefined) and resolved (NestedValue) values. Access paths are usually dot paths (a.b.c) but bracket forms can appear (a['b']).

Return Value: ProbeResult dataclass

• probe_status: "pending" | "success" | "error" | "max_attempts"
• needed_vars: list[str]
• accessed_keys: set[str]
• probe_context: NestedValue
• error_message: str | None
• accessed_locations: dict[str, tuple[str | None, int | None]]
• recommendations: list[dict] -- targets from recommend() calls encountered during probing
• rendered_markdown: str | None -- final rendered output when probe succeeds

2.3 Web Wrapper: probe_template¶

Location: vibe/web/core/state_tracker.py::probe_template

Flask wrapper -- injects g.accessed_questions tracker, updates session on errors. Called from vibe/web/interview_logic.py.

2.4 Relevance Determination¶

Two functions share the relevance algorithm, split by dependency boundary:

determine_relevant_questions_core (headless core, no Flask) Location: vibe/probing/relevance.py::determine_relevant_questions_core

Pure-logic function usable outside Flask (e.g., vibe/testing/runner.py::HeadlessInterviewRunner). Accepts explicit previous_relevant set for component stability. Now a true headless function with no Flask dependencies.

determine_relevant_questions (web wrapper) Location: vibe/web/core/state_tracker.py::determine_relevant_questions

Flask wrapper that reads g.interview["previous_relevant_questions"], adds assisted fields and appendix questions via the additional_relevant parameter, then delegates to the core logic in vibe/probing/relevance.py.

Algorithm (shared core):

1. Start with accessed_keys UNION needed_vars
2. Add component questions (if component used and question accessed/needed/previously relevant)
3. Filter to known question keys
4. Include parent questions via dependency_map (ancestor traversal)
5. Sort by original question order

Web wrapper additions (before core):

• Add appendix questions (if appendix active)
• Add assisted fields (if prompt can render with current state)

Contract Note:

• Output is question paths only (list[str]), not option-level state.
• Option-level hidden/disabled behavior should be layered on top of this contract, not inferred as a built-in probe result.

3. DATA STRUCTURES¶

3.1 NestedValue¶

Location: vibe/nested_value.py::NestedValue

Dictionary subclass supporting nested attribute access (context.a.b.c) and access tracking during probing. Key attributes: _value, _path_segment, _parent, _source (ValueSource enum), _definition. Leaf node _value holds primitive/object value -- do not assume answered values are stored as plain types at the top level.

Tracking Suppression: suppress_nv_tracking contextvars flag suppresses access recording during dict() conversion to prevent false-positive access recording (see Section 2.2 step 5).

Helper Functions: set_nested_value, get_nested_node, get_nested_value, nested_to_dict, create_nested_structure

3.2 Path Utilities¶

Location: vibe/utils/path.py

parse_path(path_str) -- Parse "items[0].name" -> ("items", 0, "name") with @lru_cache(maxsize=1024). path_to_string(parts) -- reverse.

3.3 VibeContext & Rendering Functions¶

Location: vibe/jinja/context.py

Custom Jinja2 context separating template data from internal rendering state. Internal keys (e.g., _host_template, _is_probing, meta) are stored in a _vibe_ctx dict, not in the template data tree. A single _KEY_NAMES mapping (kwarg name -> template-facing key name) is the source of truth; INTERNAL_KEYS and _DEFAULTS are derived from it. Set as context_class on all Jinja environments (vibe/jinja/env.py).

Rendering entry points (bypass Template.render()'s dict conversion to avoid mutating NestedValue): vibe_render() for Markdown/Jinja paths, make_docx_render_context() for DOCX. Web preview mode returns ChainableUndefined for missing variables (graceful degradation). VibeProbeContext subclass adds placeholder resolution.

3.4 ValueSource¶

Location: vibe/structures.py::ValueSource

Enum tracking origin of each value: UNKNOWN, INTERMEDIATE, SESSION_CONTEXT, LINKED, USER_ANSWER, PROBE_PLACEHOLDER, HOST_MAPPING, COMPONENT_DEFAULT, COMPUTED, JINJA_RESULT. Used for priority checking (don't overwrite USER_ANSWER with PROBE_PLACEHOLDER; never overwrite SESSION_CONTEXT or LINKED from form posts).

3.5 Question Definition Models¶

Location: vibe/runtime_models/definitions.py

Question definitions from config.yml are parsed into frozen Pydantic v2 models at the provider boundary via parse_question_def(). Key types: QuestionDefBase (shared fields), TextDef, BoolDef, TristateDef, EnumDef, NumberDef, DateDef, AmountDef, PeriodDef, ListDef, StructuredDef, ComputableDef, MessageDef, MultiChoiceDef. def_attr(definition, key, default) bridges Pydantic models and raw dicts. Design: frozen=True, extra="forbid", discriminated union via type field.

3.6 TemplateData¶

Location: vibe/structures.py::TemplateData

Comprehensive dataclass containing all template metadata, questions, compiled template, and validation results.

Key Fields:

Creation: vibe/templates_mgmt/provider_core.py::_load_template

3.7 ProbeResult¶

Location: vibe/probing/orchestrator.py::ProbeResult

Return value from run_probe. See Section 2.2 for fields.

4. TEMPLATE PROVIDER & VFS¶

4.1 TemplateDataProvider¶

Location: vibe/templates_mgmt/provider_core.py::TemplateDataProvider

Key Responsibilities:

• Load and parse config.yml (questions, definitions, settings)
• Resolve include: directives in config files (YAML include inheritance)
• Compile Jinja templates with custom filters/globals
• Run AST analysis to extract components, assistant blocks, appendices
• Recursively load components with dependency resolution
• Cache templates by (template_id, version_identifier)

Template Source Configuration: Each template_sources entry is either an explicit path: to a template directory, or an include: that auto-discovers subdirectories containing config.yml. Collisions raise ValueError at startup.

YAML Include (Config Inheritance): Any YAML config file can declare include: with relative paths to inherit from base configs. Recursive with circular detection, deep dict merging (including file's keys overlay base). Works across all VFS schemes.

Question Placement System: Component questions are positioned in the host template via a declarative 2-phase approach (vibe/templates_mgmt/question_placement.py):

1. Collection phase (collect_placements) -- Iterates over component instances and produces QuestionPlacement records with typed placement specs: AfterQuestion (position after a host question), FollowupOf (nest as followup), InMultichoice (gate behind multichoice option), or Standalone (append to order).
2. Resolution phase (resolve_placements) -- Single pass that writes template.questions, template.order, and template.followup_placements. Invalid placements (referencing nonexistent targets) fall through to Standalone with a validation error.

This replaces the former multi-stage mutation pipeline with an explicit data model.

Caching: Filesystem (mtime-based), Git (SHA-based immutable), Embedded (always fresh)

4.2 VirtualFileSystem¶

Location: vibe/templates_mgmt/provider_vfs.py::VirtualFileSystem

Unified API for reading files across storage backends.

URI Schemes: file://, git://<base64_repo>/<sha>/<path>, embedded://<template_id>/<path>

Key Methods: read(vpath), locate_component(component_id, host_template), load_python_module

Component Discovery: Global sources first, then relative to host (traverse up directory tree), prioritize files over directories.

See also: components.md for component system details

5. HANDLER SYSTEM (FIELD TYPES)¶

Purpose: Plugin system for field types -- each handler encapsulates widget rendering, form parsing, validation, and probe placeholders.

Location: vibe/handlers/ package

Architecture:

• DataTypeHandler abstract base class (vibe/handlers/base.py)
• QuestionRenderContext frozen dataclass (vibe/handlers/base.py) -- per-request rendering state
• HANDLER_REGISTRY -- Type name -> handler class mapping
• @register_handler(type_name) decorator for registration
• get_handler_for_definition(definition, render_context=...) -- Factory function

Schema vs. Rendering State Separation: Handler construction separates static schema (definition dict -- immutable YAML config) from per-request state (QuestionRenderContext -- frozen dataclass with internal_name, locked, question_disabled). The factory get_handler_for_definition(definition, render_context=...) passes both to DataTypeHandler.__init__().

Base Handler Methods: render_widget, process_form_data, validate_input, get_probe_placeholder

Built-in Handlers: TextHandler (text/email/password/textarea), NumberHandler, BoolHandler, TristateHandler, EnumHandler (select/radio), MultiChoiceHandler, DateHandler, AmountHandler, PeriodHandler, ListHandler, StructuredHandler, ComputableHandler, MessageHandler (note/warning/error)

6. SESSION MANAGEMENT¶

Location: vibe/web/core/session_mgmt.py

Storage: Flask session (server-side FileSystemCache in DataDirectoryManager.get_sessions_dir())

6.1 Multi-Interview Session Structure¶

The session uses a two-level structure: auth data at the session root and interview state nested under session['interviews'][key].

session = {
    # Auth-level data (shared across interviews)
    "email": str, "user_name": str, "grants": list,
    "_csrf_token": str, "devel": bool,

    # Interview data (keyed by "template_id:version_identifier")
    "interviews": {
        "my_template:abc123": {
            "session_id": str,
            "template_id": str,
            "version_id": str,
            "current_state": NestedValue,           # All answers (hierarchical)
            "previous_relevant_questions": list,
            "asked_questions": list,
            "status": "pending"|"error"|"complete",
            "error_message": str | None,
            "validation_errors": dict,
            "_generation": int,                     # Optimistic concurrency
            "_insert_cache": dict,                  # Insert-level memoization (see components.md §9)
            "linked_from": dict | None,             # Source interview metadata (linked interviews)
            "linked_bindings": list[dict],           # Data bindings from source interview
            "active_recommendations": list,          # recommend() targets for linked launches
            "assistants": dict,
            "finalized_assistants": dict,
        },
    },
}

6.2 Interview Key Derivation¶

Key format: f"{template_id}:{version_identifier}". The route_setup decorator extracts template_id and version from URL, looks up session['interviews'][key], and sets g.interview. Most code accesses state through g.interview; set session.modified = True after changes.

6.3 Key Functions¶

initialize_session(template_id, version_identifier, group_id) -- Creates interview key, initializes interview state, creates NestedValue with ValueSource.SESSION_CONTEXT. Accepts optional linked_data, linked_from, and linked_bindings for linked interview launches.

update_answers_from_form(...) -- Parses form data via handlers, updates g.interview["current_state"], validates, stores errors. Refuses to overwrite SESSION_CONTEXT or LINKED values from form posts.

7. INTERVIEW FLOW¶

7.1 Initial Request¶

GET /interview/<template_id>/@<version>/
    -> @route_setup(load_template=True)
    -> initialize_session
    -> probe_template -> run_probe -> ProbeResult
    -> determine_relevant_questions -> Ordered list
    -> Render widgets via handlers -> HTML
    -> Return full page

7.2 HTMX Auto-Submit (Field Change)¶

User changes field -> HTMX auto-submits (hx-post="/process/<template_id>/@<version>/")
    -> update_answers_from_form -> Parse + Validate
    -> probe_template -> ProbeResult
    -> Compare previous_relevant vs. new_relevant -> Identify changes
    -> Render OOB swaps (removed/new/error divs, progress bar, preview)
    -> Return HTML fragments -> HTMX swaps DOM (no page reload)

7.3 Key Routes¶

8. DEFINED TERMS SYSTEM¶

Purpose: Automatically extract defined terms, inject interactive tooltips into rendered HTML, and validate term usage for consistency.

Key File: vibe/document/defined_terms.py

Configuration: term_definitions: block in config.yml with keys enable (bool), validate (bool, requires enable), heading (str), whitelist (list). Resolved into frozen TermDefinitionsConfig dataclass.

Extraction: Two strategies -- regex-based Markdown extraction at startup (best-effort with Jinja tags), and HTML extraction at render time (accurate, post-Jinja). Host terms take precedence over component terms (first-occurrence-wins).

Inflection: _build_term_forms() generates locale-specific inflected forms (plural, definite, possessive, genitive) via LanguageRegistry. Multi-word terms inflect only the head noun.

Tooltip Injection: Wraps term occurrences in <span class="defined-term"> elements. Longest-match-first regex, skips code blocks/links/script tags.

Validation: Two checks producing ValidationIssue(severity="warning", source="term_defs"): - Reverse check (all locales): every defined term must appear in the document body. - Forward check (non-German): detects title-cased sequences that look like undefined terms.

Runs at startup (on raw template text) and at render time (on rendered HTML).

9. CONTRIBUTIONS SYSTEM¶

Purpose: Cross-cutting content routing allowing components to contribute content to named slots defined by the host template. Enables decoupled document assembly where components inject definitions, clauses, or other content into host-defined collection points.

Key Files: vibe/contributions/ package

Architecture: Two-pass marker-based resolution:

1. Pass 1 (Jinja render): {% contribute "slot_name" %}...{% endcontribute %} blocks capture content and register it with a request-scoped ContributionManager instead of emitting inline. insert_contributions("slot_name") emits an HTML comment marker at the collection point.
2. Pass 2 (post-render): resolve_markers() replaces markers with drained contribution content in document order.

Key Components:

• ContributionManager (vibe/contributions/manager.py) -- request-scoped store with hierarchical scoping (appendix subtrees drain only their own contributions)
• ContributeExtension (vibe/jinja/contribute_ext.py) -- Jinja2 extension parsing {% contribute %} blocks (multilingual aliases)
• DefinitionExtension (vibe/jinja/definition_ext.py) -- {% definition %} and {% term %} semantic markup, registers terms with DefinitionRegistry
• DefinitionRegistry (vibe/contributions/definitions.py) -- request-scoped term registry for tooltip matching
• ContributeSlotVisitor (vibe/static_analysis/visitors_extraction.py) -- AST visitor extracting slot graph for validation (contributed vs. collected slots)

Static Validation: The provider validates that every contributed slot has a corresponding collector; orphaned contributions produce warnings.

10. LINKED INTERVIEWS¶

Purpose: Allow one interview to launch another with pre-populated data via recommend() template function calls.

Data Flow:

1. Source template calls recommend("target_template_id", mapped_var=source_var) during rendering
2. AST validation (RecommendTemplateVisitor) verifies target template IDs exist
3. Session stores active_recommendations list with target IDs, mappings, and snapshot data
4. Launch via URL query parameters (recommendation_index, linked_from_template, etc.)
5. Target session receives linked_data (source state under _linked.<source_template_id>) and explicit mappings resolved against uses: declarations
6. Protection: ValueSource.LINKED values cannot be overwritten from form submissions

Session Fields: linked_from (source metadata), linked_bindings (data binding records), active_recommendations (available launch targets)

11. UTILITY MODULES¶

Server-Timing: Debug/test mode collects performance metrics via g.timings and emits Server-Timing HTTP headers.

12. FILE LOCATION INDEX¶

Core Probing¶

Data Structures¶

Template Provider & VFS¶

Handler System¶

Defined Terms¶

Contributions System¶

Behavioral Testing¶

Session, Interview & Application¶

Document Version: 6.0 Last Updated: 2026-03-31 Notes: Added contributions system (section 9), linked interviews (section 10), question placement system, ContributeSlotVisitor/RecommendTemplateVisitor, ValueSource.LINKED/JINJA_RESULT, component package split (operations/probe/relevance/cache)