Data Engine documentation

This site combines hand-written author guides with generated API reference material.

Contents

• Getting Started
  • The mental model
  • The basic workspace layout
  • Where flow module sources live
  • Your first flow
  • What the app actually does with that flow
  • A starter-style polling flow
  • Batch-oriented files
  • Running flows from Python
  • Manual, poll, and schedule at a glance
  • A few good habits early
  • Next steps
• Core Concepts
  • Flow
  • Runtime modes
  • Step
  • Saved objects
  • Batch mapping
  • Source and mirror namespaces
  • Discovery
  • Workspaces
• Configuring Flows
  • Core fields
  • Watching
  • Mirror bindings
  • Batch workflows
  • Configuring step labels and saved objects
• Authoring Flow Modules
  • Required contract
  • Step style
  • Good patterns
  • Helper modules
  • Example
  • Notebook-authored vs Python-authored modules
  • A practical authoring checklist
• Flow Methods
  • Flow(group)
  • watch(...)
  • mirror(root=...)
  • step(fn, use=None, save_as=None, label=None)
  • map(fn, use=None, save_as=None, label=None)
  • step_each(fn, use=None, save_as=None, label=None)
  • collect(extensions, root=None, recursive=False, use=None, save_as=None, label=None)
  • run_once()
  • run()
  • preview(use=None)
• Database Methods
  • context.database(...)
  • Example
  • Good patterns
  • A note on mirror vs database paths
• DuckDB Helpers
  • Import style
  • Shared conventions
  • build_dimension(...)
  • attach_dimension(...)
  • normalize_columns(...)
  • denormalize_columns(...)
  • replace_rows_by_file(...)
  • replace_rows_by_values(...)
  • compact_database(...)
  • read_rows_by_values(...)
  • read_sql(...)
  • read_table(...)
  • replace_table(...)
  • Design guidance
  • When to use direct DuckDB instead
• Recipes
  • Recipe: Mirror every workbook
  • Recipe: Filter rows and write a cleaned output
  • Recipe: Capture source metadata during processing
  • Recipe: Produce a stable latest snapshot
  • Recipe: Read selected worksheets from a multi-sheet workbook
  • Recipe: Single-file settings workflow
  • Recipe: Batch read with map(...) or step_each(...)
  • Recipe: Load into DuckDB and export a summary
  • Recipe: Use TOML workspace config
  • Recipe: Calculate business days and keep a grouped running total
  • Recipe: Offset to the next business due date
  • Recipe: Write several outputs for one source
• App Runtime and Workspaces
  • The two roots to keep in mind
  • How the app is structured
  • Authored files vs generated runtime artifacts
  • Shared workspace state
  • Local state vs workspace state
  • Control, handoff, and control requests
  • The daemon and the selected workspace
  • Workspace selection
  • Workspace provisioning
  • VS Code provisioning
  • Flow-module compilation
  • Logging and run history
  • The kill switch
  • How this affects flow authors
• FlowContext
  • What FlowContext contains
  • The three most important ideas
  • flow_name and group
  • current
  • objects
  • metadata
  • config
  • database(...)
  • source_metadata()
  • source
  • mirror
  • When source or mirror may be missing
  • Batch values
  • A practical context walkthrough
• API Reference
  • Flow Authoring
  • Core Flow Model
  • Core Primitives
  • Core Runtime Models
  • Runtime Engine
  • File Watching
  • Authoring Helpers
  • Host Surfaces
  • Application Services
• Project Map
  • Regenerating the map
  • Package Rollup
  • Largest Modules
  • Internal Stitching Points
  • Practical Mental Model
• Project Inventory