Skip to content

Conversation

@dimitri-yatsenko
Copy link
Member

No description provided.

claude and others added 29 commits December 12, 2025 03:18
…book

Add {index} directives to key database and DataJoint-specific concepts
across all major sections of the book:

- Introduction: DataJoint, computational database, data integrity,
  SQL, table tiers (manual, imported, computed, lookup), provenance,
  foreign key, schema, query algebra operators

- Concepts: database, DBMS, authentication, authorization, server-client
  architecture, embedded database, SQLite, Edgar F. Codd, relational data
  model, data model, schema-on-write/read, table, relation, tuple,
  attribute, JSON, data provenance, relational algebra/calculus, normal
  forms, Entity-Relationship Model, Peter Chen, functional dependency,
  table tiers, DAG, cascade delete

- Design: domain/entity integrity, consistency, ACID, natural key,
  composite primary key, surrogate key, UUID

- Queries: entity normalization, closure property, data independence

These index terms will now appear in the generated index page at
95-reference/index.md via the {show-index} directive.
…x-015ZtpfYQuusa2bPgFr8Y41p

Fix glossary index not working
The index page was named 'index.md' which could conflict with the
site's main index page. Renamed to 'genindex.md' (following Sphinx
convention) and added proper frontmatter title.
- Delete genindex.md (index of terms page)
- Remove all {index} directives from 9 files throughout the book

The MyST book-theme has a React bug preventing proper HTML generation,
so the index feature is being removed for now.
…x-015ZtpfYQuusa2bPgFr8Y41p

Claude/fix glossary index 015 ztpf y quusa2b pg fr8 y41p
- Relocated 012-integrity.md from 30-design to 20-concepts/04-integrity.md
- Renumbered workflows chapter (04 → 05) and pipelines chapter (05 → 06)
- Updated all cross-references in the moved chapter to point to Design section
- Updated chapter transitions to reflect new flow:
  - 03-relational-practice.ipynb now points to Data Integrity as next chapter
  - 05-workflows.md now references Data Integrity chapter
  - 04-integrity.md Next Steps now references Workflows and Design section
- Updated all files referencing renamed workflow/pipeline chapters:
  - 030-foreign-keys.ipynb
  - 050-relationships.ipynb
  - 053-master-part.ipynb
  - 010-computation.ipynb
  - 010-classic-sales.ipynb
  - 015-table.ipynb

New Concepts section flow:
1. Databases → Models → Relational → Practice → Integrity → Workflows → Pipelines

This organization places Data Integrity as a foundational concept before
introducing the Relational Workflow Model, providing better pedagogical flow.
…ntegrity-chapter-01U2NeQ3rx1Wvfiz4JF4gshL

Move Data Integrity to Concepts section
…amples

- Add detailed ACID transaction semantics for make calls (atomicity, consistency, isolation, durability)
- Include Blob Detection example showing master's responsibility to populate all parts
- Add section explaining how dependency on master implies dependency on all parts
- Include SelectDetection example demonstrating downstream table access to parts
- Improve practical guidelines with when to use and best practices
- Expand summary with key principles
- Streamline ACID transaction explanation with reference to master-part chapter
- Add dedicated section on the populate method with examples
- Simplify Blob Detection case study to avoid code duplication
- Add cross-references between chapters using seealso blocks
- Convert benefits list to table format for clarity
- Focus on workflow automation concepts unique to this chapter
- Change title from "Blob Detection Workflow" to "Blob Detection"
- Update references in Computation chapter to match
- Shorten chapter title from "Master-Part Relationships" to "Master-Part"
- Update references in Computation chapter and Data Integrity chapter
…ionship-013Zt6A82BSspcThe2yEXhZQ

Complete Master-Part Relationship chapter
- Move Design Alterations chapter to Special Topics section
- Rename "Modeling Relationships" to "Relationships"
- Incorporate Reverse Engineering content into Create Schemas chapter
- Update cross-references in Classic Sales example
…tructure-25Md7

Reorganize documentation structure and naming
- Move computation chapter to 40-operations/050-populate.ipynb
- Add new 060-orchestration.ipynb covering infrastructure concerns
- Update cross-references in 20-concepts/04-integrity.md
- Update cross-references in 30-design/015-table.ipynb
- Remove 60-computation directory

Operations section now contains:
1. Insert
2. Delete
3. Updates
4. Transactions
5. Populate (automated computation via make/populate)
6. Orchestration (infrastructure, containerization, monitoring)
…s-operations-1ZJss

Merge Computations section into Operations
Add new section explaining the three-part structure of make methods:
1. Fetch - retrieve data from upstream tables using the key
2. Compute - perform the transformation/computation
3. Insert - store results in the table and any part tables

Includes code examples and a complete example showing the pattern.
- Create new 055-make.ipynb chapter covering:
  - Input: the key dictionary from the key source
  - Three-part anatomy: fetch, compute, insert
  - Restrictions on auto-populated tables (no manual insertion)
  - Upstream-only fetching constraint (foreign key chain)
  - Three-part pattern for long-running computations
  - Transaction handling strategies

- Update 050-populate.ipynb to reference the new chapter
  instead of containing the full make anatomy inline
…ction-MOySC

Document make function anatomy in Populate
- Complete join chapter with semantic matching, foreign key relationships,
  left joins, and SQL translations
- Complete union chapter with entity type compatibility, OR logic patterns,
  and best practices
- Complete universal sets chapter covering dj.U() for extracting unique
  values, universal aggregation, and arbitrary groupings
- Complete subqueries chapter with common patterns (existence, non-existence,
  AND/OR conditions, universal quantification)
- Rewrite aggregation chapter with consistent style, clear examples using
  university database, and comprehensive SQL translations

All chapters follow consistent format with:
- Clear definitions without "powerful/fundamental" language
- Examples from university database
- SQL equivalents
- Best practices
- Practice exercises with solutions
- Cross-references to Examples section
…section-01NwAA9UPfmQB46GHkQLieEk

Complete and harmonize Queries section
Merge "Queries in Context" and "Five Query Operators" into a single
streamlined chapter. The new chapter focuses on practical pedagogy:

- Core principles (entity integrity, algebraic closure)
- Quick reference for all five operators with examples
- Comparison with SQL
- Building complex queries step by step

Removes the heavy theoretical/historical content in favor of clear,
actionable guidance for learning the query operators.
…apters-01TYwfu5hmSaYHnpDiKrtjXf

Simplify and merge query chapters
- Combined subquery patterns into the Restriction chapter since
  subqueries are just a special case of restrictions (semijoins/antijoins)
- Added Pattern 6 (Universal Quantification), Pattern 7 (Reverse Perspective),
  Self-Referencing Patterns, and Building Queries Systematically sections
- Added Summary of Patterns table for quick reference
- Added more practice exercises covering advanced subquery patterns
- Renamed "The Query Operators" to "Query Operators"
- Deleted the now-redundant 080-subqueries.ipynb chapter
…strictions-01J4bUMvEhEG9z7qjUHpLF1f

Merge subquery chapter into restrictions
@dimitri-yatsenko dimitri-yatsenko merged commit a5e0bc7 into datajoint:main Dec 12, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants