Merge pull request #267 from baszalmstra/misc/cleanup_and_docs

misc: split database and added docs

book/src/SUMMARY.md

@@ -19,3 +19,6 @@
     - [Struct Memory Kind](ch03-02-struct-memory-kind.md)
     - [Marshalling](ch03-03-marshalling.md)
     - [Hot Reloading Structs](ch03-04-hot-reloading-structs.md)
+
+- [Developer Documentation](ch04-00-developer-docs.md)
+    - [Salsa](ch04-01-salsa.md)

book/src/ch04-00-developer-docs.md

@@ -0,0 +1,7 @@
+# Developer documentation
+
+These chapters provide some insight in the design of the Mun compiler and 
+language server.
+
+* [Salsa](ch04-01-salsa.md) provides some information about Salsa and how we use 
+it.

book/src/ch04-01-salsa.md

@@ -0,0 +1,102 @@
+# Salsa
+
+The Mun backend makes extensive use of 
+[Salsa](https://github.com/salsa-rs/salsa) to perform on-demand, incremental 
+computations. `salsa` lets you define your program as a set of *queries*. Every 
+query is used like a function `K -> V` that maps from some key `K` to a value of
+`V`. Changing the input `K` also invalidates the result of computing `K -> V`. 
+However, if the result `V` does not change after changing `K` the invalidation 
+is not propagated to computations that use `V` as an input. This enables Mun to
+cache a lot of computations between compilations, enabling 
+very fast rebuilding on incremental changes.
+
+> For more in depth information on Salsa please refer to the 
+>[Salsa repository](https://github.com/salsa-rs/salsa) or the
+>[Salsa book](https://salsa-rs.github.io/salsa/)
+
+## Databases
+
+Queries are grouped together in so-called *databases* defined as traits. 
+Database traits can have super traits that enable them to build upon other 
+queries.
+
+![Mun Salsa Database super trait relations](./ch04-01-salsa-super-traits.png) 
+
+<span class="caption">Figure 4-1: Super trait relations between Salsa databases</span>
+
+> This design is heavily inspired by 
+>[Rust Analyzer](https://github.com/rust-analyzer/rust-analyzer).
+
+### `SourceDatabase`
+
+The `SourceDatabase` provides queries and inputs that form the basis of all
+ compilation because it contains the original source text. 
+
+The `SourceDatabase` holds `SourceRoot`s which are groupings of files. 
+`SourceRoot`s define the hierarchy of files relative to a certain root. For the 
+system it's not interesting where the files come from and how they are actually 
+organized on disk. `SourceRoot`s provide an abstraction over the filesystem that
+ provides just enough information for the backend to function.
+
+Another relevant query is the `line_index` query which provides line and column 
+information for characters in a source file. This is used for instance when 
+emitting human-readable diagnostic messages with line and column numbers.
+
+### `AstDatabase`
+
+The `AstDatabase` provides syntax tree transformations on top of the source
+text. The `parse` function returns the 
+[Concrete Syntax Tree](https://en.wikipedia.org/wiki/Parse_tree) (CST) 
+of a file provided by the `mun_syntax` crate. Any errors related to parsing are 
+stored in the tree itself. 
+
+As an example of where Salsa helps Mun achieve faster incremental compilation: 
+changing a single character in a file will also change the CST which in turn
+will invalidate a lot of computations. However, most changes occur within blocks 
+(like functions or structs). To not invalidate everything in the changed file,
+another query transforms the CST to an `AstIdMap` which assigns ids to all  
+top-level items in a file. This enables referring to those ids in later stages 
+of compilation which will change much less often.
+
+### `InternDatabase`
+
+The `InternDatase` links certain locations in the syntax tree to ids. For
+instance, it associates a function syntax node with a `FunctionId`. This id
+is then used everywhere in code to refer to the specific function instance. The 
+same happens for other items like structs. The purpose of this *interning* is 
+again to allow better incremental compilation for queries that only need to 
+refer to certain constructs without having to refer to the volatile syntax tree
+elements. 
+
+### `DefDatabase`
+
+The `DefDatabase` provides definitions extracted from the syntax trees. It 
+provides desugared or lowered versions of language constructs that enable other
+queries to query specific information for these constructs. Extracting this data
+again enables faster incremental compilation because if the extracted data does 
+not change dependent queries also will not change.
+
+### `HirDatabase`
+
+The `HirDatabase` resolves all types and names from the definitions. It also 
+performs type inferencing. The resulting High-Level Representation (HIR) is the
+input for the LLVM Intermediate Representation (LLVM IR). The HIR data 
+structures can also contain errors, but only if there are no errors present will 
+generating IR be valid.
+
+### `CodeGenDatabase`
+
+Finally, the `CodeGenDatabase` provides queries to construct assemblies from 
+groups of files. Assemblies are the final product of the Mun compiler. They are 
+the binaries that can be executed by the runtime. This stage is where Salsa 
+really shines; only if something actually changed in the HIR will an assembly be 
+created. 
+
+## Language Server
+
+The language server uses the same backend as the compiler. Whereas the compiler 
+creates assemblies the language server queries the different databases to 
+provide a user with code intelligence. It also uses the exact same diagnostics  
+paths as the compiler, which means there is never a mismatch between the   
+compiler and the language server - and vice versa. If compiling your code 
+results in an error it will also be immediately visible in your IDE. 

crates/mun_codegen/src/mock.rs

@@ -9,6 +9,8 @@ use std::sync::Arc;
 /// A mock implementation of the IR database. It can be used to set up a simple test case.
 #[salsa::database(
     hir::SourceDatabaseStorage,
+    hir::AstDatabaseStorage,
+    hir::InternDatabaseStorage,
     hir::DefDatabaseStorage,
     hir::HirDatabaseStorage,
     CodeGenDatabaseStorage
@@ -28,6 +30,12 @@ impl salsa::Database for MockDatabase {
     }
 }
 
+impl hir::Upcast<dyn hir::AstDatabase> for MockDatabase {
+    fn upcast(&self) -> &dyn hir::AstDatabase {
+        &*self
+    }
+}
+
 impl hir::Upcast<dyn hir::SourceDatabase> for MockDatabase {
     fn upcast(&self) -> &dyn hir::SourceDatabase {
         &*self

crates/mun_compiler/src/db.rs

@@ -5,6 +5,8 @@ use mun_hir::{salsa, HirDatabase, Upcast};
 /// A compiler database is a salsa database that enables increment compilation.
 #[salsa::database(
     mun_hir::SourceDatabaseStorage,
+    mun_hir::InternDatabaseStorage,
+    mun_hir::AstDatabaseStorage,
     mun_hir::DefDatabaseStorage,
     mun_hir::HirDatabaseStorage,
     CodeGenDatabaseStorage
@@ -13,6 +15,12 @@ pub struct CompilerDatabase {
     storage: salsa::Storage<Self>,
 }
 
+impl Upcast<dyn mun_hir::AstDatabase> for CompilerDatabase {
+    fn upcast(&self) -> &dyn mun_hir::AstDatabase {
+        &*self
+    }
+}
+
 impl Upcast<dyn mun_hir::SourceDatabase> for CompilerDatabase {
     fn upcast(&self) -> &dyn mun_hir::SourceDatabase {
         &*self

crates/mun_compiler/src/driver.rs

@@ -6,7 +6,9 @@ use crate::{
     PathOrInline, RelativePath,
 };
 use mun_codegen::{Assembly, CodeGenDatabase};
-use mun_hir::{DiagnosticSink, FileId, RelativePathBuf, SourceDatabase, SourceRoot, SourceRootId};
+use mun_hir::{
+    AstDatabase, DiagnosticSink, FileId, RelativePathBuf, SourceDatabase, SourceRoot, SourceRootId,
+};
 
 use std::{path::PathBuf, sync::Arc};
 

crates/mun_hir/src/db.rs

@@ -24,8 +24,7 @@ pub trait Upcast<T: ?Sized> {
     fn upcast(&self) -> &T;
 }
 
-/// Database which stores all significant input facts: source code and project model. Everything
-/// else in rust-analyzer is derived from these queries.
+/// Database which stores all significant input facts: source code and project model.
 #[salsa::query_group(SourceDatabaseStorage)]
 pub trait SourceDatabase: salsa::Database {
     /// Text of the file.
@@ -36,10 +35,6 @@ pub trait SourceDatabase: salsa::Database {
     #[salsa::input]
     fn file_relative_path(&self, file_id: FileId) -> RelativePathBuf;
 
-    /// Parses the file into the syntax tree.
-    #[salsa::invoke(parse_query)]
-    fn parse(&self, file_id: FileId) -> Parse<ast::SourceFile>;
-
     /// Source root of a file
     #[salsa::input]
     fn file_source_root(&self, file_id: FileId) -> SourceRootId;
@@ -53,13 +48,34 @@ pub trait SourceDatabase: salsa::Database {
     fn line_index(&self, file_id: FileId) -> Arc<LineIndex>;
 }
 
-#[salsa::query_group(DefDatabaseStorage)]
-pub trait DefDatabase: SourceDatabase + Upcast<dyn SourceDatabase> {
+/// The `AstDatabase` provides queries that transform text from the `SourceDatabase` into an
+/// Abstract Syntax Tree (AST).
+#[salsa::query_group(AstDatabaseStorage)]
+pub trait AstDatabase: SourceDatabase {
+    /// Parses the file into the syntax tree.
+    #[salsa::invoke(parse_query)]
+    fn parse(&self, file_id: FileId) -> Parse<ast::SourceFile>;
+
     /// Returns the top level AST items of a file
     #[salsa::invoke(crate::source_id::AstIdMap::ast_id_map_query)]
     fn ast_id_map(&self, file_id: FileId) -> Arc<AstIdMap>;
+}
+
+/// The `InternDatabase` maps certain datastructures to ids. These ids refer to instances of
+/// concepts like a `Function`, `Struct` or `TypeAlias` in a semi-stable way.
+#[salsa::query_group(InternDatabaseStorage)]
+pub trait InternDatabase: SourceDatabase {
+    #[salsa::interned]
+    fn intern_function(&self, loc: ids::ItemLoc<ast::FunctionDef>) -> ids::FunctionId;
+    #[salsa::interned]
+    fn intern_struct(&self, loc: ids::ItemLoc<ast::StructDef>) -> ids::StructId;
+    #[salsa::interned]
+    fn intern_type_alias(&self, loc: ids::ItemLoc<ast::TypeAliasDef>) -> ids::TypeAliasId;
+}
 
-    /// Returns the raw items of a file
+#[salsa::query_group(DefDatabaseStorage)]
+pub trait DefDatabase: InternDatabase + AstDatabase + Upcast<dyn AstDatabase> {
+    /// Returns the top level items of a file.
     #[salsa::invoke(RawItems::raw_file_items_query)]
     fn raw_items(&self, file_id: FileId) -> Arc<RawItems>;
 
@@ -75,18 +91,6 @@ pub trait DefDatabase: SourceDatabase + Upcast<dyn SourceDatabase> {
     /// Returns the module data of the specified file
     #[salsa::invoke(crate::code_model::ModuleData::module_data_query)]
     fn module_data(&self, file_id: FileId) -> Arc<ModuleData>;
-
-    /// Interns a function definition
-    #[salsa::interned]
-    fn intern_function(&self, loc: ids::ItemLoc<ast::FunctionDef>) -> ids::FunctionId;
-
-    /// Interns a struct definition
-    #[salsa::interned]
-    fn intern_struct(&self, loc: ids::ItemLoc<ast::StructDef>) -> ids::StructId;
-
-    /// Interns a type alias definition
-    #[salsa::interned]
-    fn intern_type_alias(&self, loc: ids::ItemLoc<ast::TypeAliasDef>) -> ids::TypeAliasId;
 }
 
 #[salsa::query_group(HirDatabaseStorage)]
@@ -131,7 +135,7 @@ pub trait HirDatabase: DefDatabase + Upcast<dyn DefDatabase> {
     ) -> (Arc<crate::expr::Body>, Arc<crate::expr::BodySourceMap>);
 }
 
-fn parse_query(db: &dyn SourceDatabase, file_id: FileId) -> Parse<SourceFile> {
+fn parse_query(db: &dyn AstDatabase, file_id: FileId) -> Parse<SourceFile> {
     let text = db.file_text(file_id);
     SourceFile::parse(&*text)
 }

crates/mun_hir/src/expr/validator/tests.rs

@@ -1,7 +1,10 @@
-use crate::db::{SourceDatabase, Upcast};
-use crate::expr::validator::{ExprValidator, TypeAliasValidator};
 use crate::{
-    diagnostics::DiagnosticSink, ids::LocationCtx, mock::MockDatabase, Function, TypeAlias,
+    db::{AstDatabase, Upcast},
+    diagnostics::DiagnosticSink,
+    expr::validator::{ExprValidator, TypeAliasValidator},
+    ids::LocationCtx,
+    mock::MockDatabase,
+    Function, TypeAlias,
 };
 use mun_syntax::{ast, AstNode};
 use std::fmt::Write;

crates/mun_hir/src/in_file.rs

@@ -1,4 +1,4 @@
-use crate::{FileId, SourceDatabase};
+use crate::{AstDatabase, FileId};
 use mun_syntax::SyntaxNode;
 
 /// `InFile<T>` stores a value of `T` inside a particular file/syntax tree.
@@ -30,7 +30,7 @@ impl<T> InFile<T> {
     pub fn as_ref(&self) -> InFile<&T> {
         self.with_value(&self.value)
     }
-    pub fn file_syntax(&self, db: &dyn SourceDatabase) -> SyntaxNode {
+    pub fn file_syntax(&self, db: &dyn AstDatabase) -> SyntaxNode {
         db.parse(self.file_id).syntax_node()
     }
 }

crates/mun_hir/src/lib.rs

@@ -43,7 +43,8 @@ pub use relative_path::{RelativePath, RelativePathBuf};
 pub use crate::{
     builtin_type::{FloatBitness, IntBitness, Signedness},
     db::{
-        DefDatabase, DefDatabaseStorage, HirDatabase, HirDatabaseStorage, SourceDatabase,
+        AstDatabase, AstDatabaseStorage, DefDatabase, DefDatabaseStorage, HirDatabase,
+        HirDatabaseStorage, InternDatabase, InternDatabaseStorage, SourceDatabase,
         SourceDatabaseStorage, Upcast,
     },
     diagnostics::{Diagnostic, DiagnosticSink},

crates/mun_hir/src/mock.rs

@@ -1,4 +1,4 @@
-use crate::db::SourceDatabase;
+use crate::db::{AstDatabase, SourceDatabase};
 use crate::db::{HirDatabase, Upcast};
 use crate::input::{SourceRoot, SourceRootId};
 use crate::{DefDatabase, FileId, RelativePathBuf};
@@ -9,6 +9,8 @@ use std::sync::Arc;
 /// A mock implementation of the IR database. It can be used to set up a simple test case.
 #[salsa::database(
     crate::SourceDatabaseStorage,
+    crate::AstDatabaseStorage,
+    crate::InternDatabaseStorage,
     crate::DefDatabaseStorage,
     crate::HirDatabaseStorage
 )]
@@ -27,6 +29,12 @@ impl salsa::Database for MockDatabase {
     }
 }
 
+impl Upcast<dyn AstDatabase> for MockDatabase {
+    fn upcast(&self) -> &dyn AstDatabase {
+        &*self
+    }
+}
+
 impl Upcast<dyn DefDatabase> for MockDatabase {
     fn upcast(&self) -> &dyn DefDatabase {
         &*self

crates/mun_hir/src/source_id.rs

@@ -1,10 +1,16 @@
-use crate::arena::{Arena, Idx};
-use crate::in_file::InFile;
-use crate::{db::DefDatabase, FileId};
+use crate::{
+    arena::{Arena, Idx},
+    db::AstDatabase,
+    db::DefDatabase,
+    in_file::InFile,
+    FileId,
+};
 use mun_syntax::{ast, AstNode, AstPtr, SyntaxNode, SyntaxNodePtr};
-use std::hash::{Hash, Hasher};
-use std::marker::PhantomData;
-use std::sync::Arc;
+use std::{
+    hash::{Hash, Hasher},
+    marker::PhantomData,
+    sync::Arc,
+};
 
 /// `AstId` points to an AST node in any file.
 ///
@@ -60,20 +66,11 @@ pub struct AstIdMap {
 type ErasedFileAstId = Idx<SyntaxNodePtr>;
 
 impl AstIdMap {
-    pub(crate) fn ast_id_map_query(db: &dyn DefDatabase, file_id: FileId) -> Arc<AstIdMap> {
+    pub(crate) fn ast_id_map_query(db: &dyn AstDatabase, file_id: FileId) -> Arc<AstIdMap> {
         let map = AstIdMap::from_source(&db.parse(file_id).tree().syntax());
         Arc::new(map)
     }
 
-    pub(crate) fn file_item_query(
-        db: &dyn DefDatabase,
-        file_id: FileId,
-        ast_id: ErasedFileAstId,
-    ) -> SyntaxNode {
-        let node = db.parse(file_id);
-        db.ast_id_map(file_id).arena[ast_id].to_node(&node.tree().syntax())
-    }
-
     pub(crate) fn ast_id<N: AstNode>(&self, item: &N) -> FileAstId<N> {
         let ptr = SyntaxNodePtr::new(item.syntax());
         let raw = match self.arena.iter().find(|(_id, i)| **i == ptr) {

crates/mun_hir/src/ty/tests.rs

@@ -1,12 +1,9 @@
-use crate::db::SourceDatabase;
-use crate::diagnostics::DiagnosticSink;
-use crate::expr::BodySourceMap;
-use crate::ids::LocationCtx;
-use crate::mock::MockDatabase;
-use crate::{Function, HirDisplay, InferenceResult, TypeAlias};
+use crate::{
+    db::AstDatabase, diagnostics::DiagnosticSink, expr::BodySourceMap, ids::LocationCtx,
+    mock::MockDatabase, Function, HirDisplay, InferenceResult, TypeAlias,
+};
 use mun_syntax::{ast, AstNode};
-use std::fmt::Write;
-use std::sync::Arc;
+use std::{fmt::Write, sync::Arc};
 
 #[test]
 fn comparison_not_implemented_for_struct() {