Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

21364: Improves entity persistence: works as single transactional files as well as performance improvements, MINOR #315

Open
wants to merge 52 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
041395f
22194: Begins implementing data structures for transactional entity s…
howsohazard Nov 19, 2024
8e55ba0
Merge branch 'main' into 22194-transactional-entity-storage
howsohazard Nov 20, 2024
79ccd9b
22194: Fixes compilation issues
howsohazard Nov 20, 2024
382e442
22194: More implementation progress
howsohazard Nov 20, 2024
c89b151
22194: Simplifie and removes some shared_ptr usage
howsohazard Nov 20, 2024
1f40728
22194: More implementation progress
howsohazard Nov 20, 2024
ede46be
22194: More implementation
howsohazard Nov 22, 2024
863bc95
22194: More implementation
howsohazard Nov 22, 2024
aefff58
22194: More write implementation
howsohazard Nov 22, 2024
e144823
22194: More implementation progress
howsohazard Nov 23, 2024
eb2ae0c
22194: Begins implementing versions for flattened amlg files
howsohazard Nov 23, 2024
3801d41
22194: More implementation
howsohazard Nov 25, 2024
2dce4bc
22194: Bug fixes, begins creating test
howsohazard Nov 25, 2024
ea064c0
Merge branch 'main' into 22194-transactional-entity-storage
howsohazard Nov 25, 2024
37dfb52
22194: Renames a few variables in prep for richer permissions
howsohazard Nov 25, 2024
90ae12e
Merge branch 'main' into 22194-transactional-entity-storage
howsohazard Nov 25, 2024
a694514
22194: Updates todos
howsohazard Nov 25, 2024
1217544
22194: Implements part of entity permissions
howsohazard Nov 26, 2024
5402534
22194: More progress
howsohazard Nov 26, 2024
d5c65aa
Merge branch 'main' into 22194-transactional-entity-storage
howsohazard Nov 26, 2024
87ff798
22194: More implementation progress
howsohazard Nov 26, 2024
3e4d020
22194: More implementation progress
howsohazard Nov 27, 2024
0e1b72f
22194: Updates todos
howsohazard Nov 27, 2024
fd19696
22194: More implementation progress
howsohazard Nov 27, 2024
90e7cbc
22194: Variable rename
howsohazard Nov 27, 2024
1462ab5
Merge branch 'main' into 22194-transactional-entity-storage
howsohazard Nov 28, 2024
7767f7f
22194: More implementation
howsohazard Nov 28, 2024
f8f8393
22194: More implementation progress
howsohazard Nov 29, 2024
6f16cc3
22194: Finishes main implementation
howsohazard Nov 29, 2024
712f7e1
21364: Begins improving caml storage, but introduces bug
howsohazard Nov 30, 2024
c0bcb22
21364: Fixes bugs
howsohazard Nov 30, 2024
81404ca
21364: Fixes bugs with clang
howsohazard Nov 30, 2024
034dba9
21364: BinaryPacking API cleanup
howsohazard Dec 1, 2024
63e601c
21364: Minor code cleanup
howsohazard Dec 2, 2024
13e8fe5
21364: Rearranges some code
howsohazard Dec 2, 2024
8717d48
21364: Finishes efforts
howsohazard Dec 2, 2024
451cbf9
21364: Fixes some gcc complaints
howsohazard Dec 2, 2024
39a88f9
21364: Fixes another gcc complaint
howsohazard Dec 2, 2024
854bbd2
21364: Tightens version read buffers
howsohazard Dec 2, 2024
8e038c3
Merge branch 'main' into 21364-caml-improvements
howsohazard Dec 2, 2024
ebdef0b
Merge branch 'main' into 21364-caml-improvements
howsohazard Dec 4, 2024
66558cb
Merge branch 'main' into 21364-caml-improvements
howsohazard Dec 4, 2024
3ea7485
Merge branch '21364-caml-improvements' of https://github.com/howsoai/…
howsohazard Dec 4, 2024
1b62f47
21364: Fixes bug with clearing persistence
howsohazard Dec 5, 2024
61b36f1
Merge branch 'main' into 21364-caml-improvements
howsohazard Dec 5, 2024
419b5b8
Merge branch 'main' into 21364-caml-improvements
howsohazard Dec 6, 2024
48a904e
21364: Fixes inconsistency of not removing flattened entity files
howsohazard Dec 6, 2024
bd4072b
21364: Fixes bugs
howsohazard Dec 6, 2024
0852ee2
Merge branch 'main' into 21364-caml-improvements
howsohazard Dec 10, 2024
e8598c1
21364: Fixes edge cases with re-storing persistent flattened files
howsohazard Dec 10, 2024
cf255f8
Merge branch 'main' into 21364-caml-improvements
howsohazard Dec 12, 2024
3b6ee03
Merge branch '21364-caml-improvements' of https://github.com/howsoai/…
howsohazard Dec 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,7 @@ <h2>File I/O</h2>
<div class='td1'><span class="parameter">flatten<span></div><div class='td2'>If true, then will attempt to flatten all contained entities into one executable object and thus one file.</div><br />
<div class='td1'><span class="parameter">parallel_create<span></div><div class='td2'>If true, will attempt use concurrency to store and load entities in parallel.</div><br />
<div class='td1'><span class="parameter">execute_on_load<span></div><div class='td2'>If true, will execute the code upon load, which is required when entities are stored using flatten in order to create all of the entity structures.</div><br />
<div class='td1'><span class="parameter">require_version_compatibility<span></div><div class='td2'>If true, will fail on a load if the version of Amalgam is not compatible with the file version.</div><br />

File formats supported are amlg, json, yaml, csv, and caml; anything not in this list will be loaded as a binary string. Note that loading from a non-'.amlg' extension will only ever provide lists, assocs, numbers, and strings.

Expand Down
4 changes: 2 additions & 2 deletions docs/language.js
Original file line number Diff line number Diff line change
Expand Up @@ -1165,11 +1165,11 @@ var data = [
},

{
"parameter" : "flatten_entity id entity [bool include_rand_seeds] [bool parallel_create]",
"parameter" : "flatten_entity id entity [bool include_rand_seeds] [bool parallel_create] [bool include_version]",
"output" : "*",
"permissions" : "e",
"new value" : "new",
"description" : "Evaluates to code that, if called, would completely reproduce the entity specified by id, as well as all contained entities. If include_rand_seeds is true, its default, it will include all entities' random seeds. If parallel_create is true, then the creates will be performed with parallel markers as appropriate for each group of contained entities. The code returned accepts two parameters, create_new_entity, which defaults to true, and new_entity, which defaults to null. If create_new_entity is true, then it will create a new entity with id specified by new_entity, where null will create an unnamed entity. If create_new_entity is false, then it will overwrite the current entity's code and create all contained entities.",
"description" : "Evaluates to code that, if called, would completely reproduce the entity specified by id, as well as all contained entities. If include_rand_seeds is true, its default, it will include all entities' random seeds. If parallel_create is true, then the creates will be performed with parallel markers as appropriate for each group of contained entities. If include_version is true, it will include a comment on the top node that is the current version of the Amalgam interpreter, which can be used for validating interoperability when loading code. The code returned accepts two parameters, create_new_entity, which defaults to true, and new_entity, which defaults to null. If create_new_entity is true, then it will create a new entity with id specified by new_entity, where null will create an unnamed entity. If create_new_entity is false, then it will overwrite the current entity's code and create all contained entities.",
"example" : "(create_entities \"FlattenTest\" (lambda\n (parallel ##a (rand) )\n))\n(let (assoc fe (flatten_entity \"FlattenTest\"))\n (print fe)\n (print (flatten_entity (call fe)))\n (print (difference_entities \"FlattenTest\" (call fe)))\n (call fe (assoc create_new_entity (false) new_entity \"new_entity_name\")) \n)"
},

Expand Down
9 changes: 5 additions & 4 deletions src/Amalgam/AmalgamMain.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -235,14 +235,15 @@ PLATFORM_MAIN_CONSOLE
{
//run the standard amlg command line interface
EntityExternalInterface::LoadEntityStatus status;
AssetManager::AssetParameters asset_params(amlg_file_to_run, "", true);
std::string file_type = "";
AssetManager::AssetParametersRef asset_params
= std::make_shared<AssetManager::AssetParameters>(amlg_file_to_run, "", true);

Entity *entity = asset_manager.LoadEntityFromResource(asset_params, false, random_seed, nullptr, status);

if(!status.loaded)
return 1;

asset_manager.SetRootPermission(entity, true);
asset_manager.SetEntityPermissions(entity, EntityPermissions::AllPermissions());

PrintListener *print_listener = nullptr;
std::vector<EntityWriteListener *> write_listeners;
Expand All @@ -258,7 +259,7 @@ PLATFORM_MAIN_CONSOLE

if(write_log_filename != "")
{
EntityWriteListener *write_log = new EntityWriteListener(entity, false, write_log_filename);
EntityWriteListener *write_log = new EntityWriteListener(entity, false, false, false, write_log_filename);
write_listeners.push_back(write_log);
}

Expand Down
Loading
Loading