Refactoring Holons Zome Module Structure

[evomimic]

Summary

Explorations conducted as part of Issue #188 have led to a number of key insights and conclusions regarding how to restructure the MAP Holons Zome modules.

Summary of Key Architectural Findings

1. Separation of Concerns

• The shared_objects_layer and reference_layer are clearly distinct:
  • Shared Objects Layer: Manages the underlying shared state (e.g., HolonSpaceManager, HolonCache, Nursery) and provides internal APIs for accessing and manipulating this state.
  • Reference Layer: Exposes high-level APIs for interacting with holons and relationships without duplicating or directly exposing shared state.
• Maintaining distinct API modules for the two layers reinforces their respective responsibilities:
  • shared_objects_layer::api provides internal APIs for shared state management.
  • reference_layer::api offers external-facing traits like HolonReadable and HolonWritable.

---

2. HolonReference as a Central Abstraction

• HolonReference, along with its variants (StagedReference, SmartReference), mediates all access to holons in the shared objects layer.
• This encapsulation:
  • Prevents direct access to holons or their state outside the shared objects layer.
  • Encourages consistency by ensuring all operations on holons go through a controlled interface.
  • Aligns with the principle of minimizing redundant data by ensuring each holon exists only once in memory.

---

3. Consolidation of Implementation Logic

• Merging the reference_layer::implementation and shared_objects_layer::implementation simplifies the architecture by:
  • Avoiding unnecessary delegation or duplication.
  • Centralizing shared state and reference management into a single implementation layer.
• This approach reflects the reality that these two layers are tightly coupled in terms of their implementation details.

---

4. HolonsContext and Context Behavior

• HolonsContext is a pivotal structure for managing shared state during a request. Moving its definition to the shared_objects_layer::implementation aligns with its role in managing shared state.
• Defining the HolonsContextBehavior trait in shared_objects_layer::api achieves:
  • Encapsulation of HolonsContext internals, exposing only the necessary methods for upper layers.
  • Decoupling of reference_layer from the concrete HolonsContext implementation, preserving modularity and flexibility.

---

5. Managing Relationships

• The use of RelationshipMap and HolonCollection in the definition of Holon (via its relationship_map field) was the key factor driving the decision to merge the reference_layer and shared_objects_layer implementation modules. Holon clearly belonged to SharedObjects and HolonCollection's dependency on HolonReference pretty much pulled everything else down into the shared objects implementation layer.
• The RelationshipMap and HolonCollection structures, moved to the shared objects layer, now operate seamlessly with HolonReference.
• This arrangement ensures:
  • Centralized control over holon relationships without duplicating state.
  • Lazy-loading and resolution of relationships as needed, mediated by HolonSpaceManager.

---

6. Stateless Server Design

• The design supports the stateless server principle by:
  • Re-initializing HolonsContext on each request in the dance_layer.
  • Using HolonsContextBehavior to abstract shared state interactions, making the context easily reconstructible.

---

7. Testing and Extensibility

• The use of traits like HolonsContextBehavior enables:
  • Mocking for unit tests, ensuring that higher-layer APIs can be tested independently of the shared objects layer.
  • Potential future extensions of HolonsContext without breaking the reference layer.

---

8. Simplification of Layer Responsibilities

• Once it became clear that the implementation layers of reference and shared objects needed to be merged, I considered whether the reference_layer was still necessary. The conclusion was that it plays an important role in:
  • Defining high-level behavior (via HolonReadable and HolonWritable).
  • Ensuring upper layers only access Holons via HolonReference (and its variants) rather than attempting to access Holon directly
  • Abstracting shared objects layer details for upper layers like dance_layer or query_layer.

---

Key Architectural Principles Reinforced

1. Encapsulation:
   • Shared state (Holon, HolonReference, etc.) remains fully encapsulated in the shared objects layer, preventing direct access from upper layers.
2. Single Responsibility:
   • The shared objects layer focuses on state management and resolution.
   • The reference layer focuses on exposing high-level behaviors for holons and relationships.
3. Minimized Duplication:
   • Avoiding redundant APIs or data copies across layers ensures efficient memory usage and reduces complexity.
4. Flexibility and Modularity:
   • Using traits like HolonsContextBehavior decouples the reference layer from specific implementations, enabling future refactoring or extension.

---

Challenges Addressed

1. Circular Dependencies:
   • Resolved by carefully placing traits and implementations in their appropriate layers.
   • HolonsContextBehavior plays a pivotal role in breaking dependency cycles.
2. Layer Blurring:
   • Consolidating implementation logic avoids unnecessary complexity while maintaining distinct APIs for shared objects and reference layers.
3. Developer Experience:
   • The use of HolonReference and its traits encourages developers to follow the intended access patterns without requiring significant boilerplate or complexity.

---

Final Thoughts

This conversation has clarified the architecture significantly, resulting in:

• A clean separation of responsibilities between layers.
• A unified implementation layer for shared state and reference management.
• Simplified access patterns through traits and controlled abstractions.

[evomimic]

After working through this re-factoring, some general guidelines have emerged that can be used as a guide to both the current layering and the placement of new modules in the appropriate layer.

General Heuristic for Struct Placement in MAP's Layered

These guidelines help maintain modularity, avoid circular dependencies, and ensure clean separation of concerns.

1. Define Traits in API Layers

• Traits define the external-facing behavior of modules.
• They act as contracts, decoupling consumers from concrete implementations.

2. Place Structs/Enums in API Layers if:

• They appear in the public signatures of the traits (e.g., as parameters or return types).
• They are lightweight, data-only structures without significant encapsulated behavior.
• Encapsulation within a trait would add unnecessary complexity or indirection.

3. Encapsulate Structs/Enums Behind Traits if:

• They have behavior likely to change independently of their consumers.
• Multiple implementations of the same behavior are required, or polymorphism is needed.
• Encapsulation significantly reduces dependency surface area or enforces stricter access control.

4. Keep Concrete Implementations in Implementation Modules

• Implementation modules depend on API modules, but the reverse should never occur.
• Place concrete implementations of traits or complex logic in the implementation module.

5. Prefer Narrow Interfaces

• If a struct is directly exposed in a trait's public signature, ensure its interface is minimal and focused only on what consumers need.
• Avoid exposing internal implementation details through the API.

6. Use Re-exports Judiciously

• Re-export structs or enums from API modules only when they form part of the essential contract of the module.
• Avoid re-exporting large parts of implementation modules to prevent exposing internal logic.

7. Ensure Uni-directional Dependency Flow

• Design API and implementation modules such that the dependency flow adheres to dependency inversion principles.
• Upper layers (e.g., reference layer) depend only on the API, while lower layers (e.g., shared objects implementation) depend on both their own API and higher layers.

---

Example Application

Shared Objects Layer

• API Module:
  • Contains traits like NurseryAccess and HolonsCacheAccess that define behaviors exposed to other layers.
• Implementation Module:
  • Contains concrete implementations of these traits and any complex logic or helper structures required internally.

Reference Layer

• Structs like HolonReference, SmartReference, and StagedReference belong in the reference layer.
  • These structs serve as lightweight abstractions for accessing staged or committed holons.
  • They are part of the reference layer's public API and should not encapsulate significant behavior.