Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node syncing #8475

Merged
merged 96 commits into from
Mar 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
e3b20fa
syncing part 1
koenvanderveen Feb 9, 2024
b074883
-
koenvanderveen Feb 13, 2024
81d567a
add get_dependents
eelcovdw Feb 13, 2024
fe3173c
Merge branch 'node-state-sync' into eelco/sync-dependencies
eelcovdw Feb 13, 2024
4d5d9dc
update get_dependencies
eelcovdw Feb 14, 2024
536c59e
cleanup
eelcovdw Feb 14, 2024
7f02029
-
koenvanderveen Feb 14, 2024
38e414c
Merge branch 'node-state-sync' into eelco/sync-dependencies
eelcovdw Feb 14, 2024
643d91b
fix input policy before approve
eelcovdw Feb 14, 2024
6319a5e
Merge pull request #8482 from OpenMined/eelco/sync-dependencies
eelcovdw Feb 14, 2024
86babe2
debugging
koenvanderveen Feb 16, 2024
6cc872a
debugging
koenvanderveen Feb 16, 2024
5e3c31f
-
koenvanderveen Feb 16, 2024
578ef1b
added diff classes
teo-milea Feb 16, 2024
616ce26
Merge branch 'node-state-sync' into node-state-sync-diff
teo-milea Feb 16, 2024
b6b3a63
added resolve
teo-milea Feb 16, 2024
292f1d4
Merge remote-tracking branch 'refs/remotes/origin/node-state-sync-dif…
teo-milea Feb 16, 2024
bfec892
fixing checking changes
teo-milea Feb 16, 2024
42eafc6
split sync state func
eelcovdw Feb 16, 2024
b0f4f61
manual merge syncing
eelcovdw Feb 16, 2024
faf654b
merge
eelcovdw Feb 16, 2024
948a392
add notebook from video
eelcovdw Feb 16, 2024
6a5959f
minor syncing fixes, fix get_dependencies
eelcovdw Feb 19, 2024
14bc9b4
add sync service + hierarchy
eelcovdw Feb 20, 2024
67d9dba
added attr for new obj
teo-milea Feb 20, 2024
04bfe2e
hack for resizing tables
teo-milea Feb 20, 2024
efa686d
fix merge
teo-milea Feb 20, 2024
83c48d7
moved repr
teo-milea Feb 20, 2024
39b8574
fixing permissions for node syncing
koenvanderveen Feb 20, 2024
01ddf14
-
koenvanderveen Feb 20, 2024
8c3a5b5
Merge branch 'node-state-sync-diff' of github.com:OpenMined/PySyft in…
koenvanderveen Feb 20, 2024
33fc2d2
merge
koenvanderveen Feb 20, 2024
8314950
hierarchy fixes
eelcovdw Feb 21, 2024
e4b5817
merge
eelcovdw Feb 21, 2024
cbfa631
fix hierarchy actionobject LineageID
eelcovdw Feb 21, 2024
66358e3
lint
koenvanderveen Feb 21, 2024
f613bdd
merge
koenvanderveen Feb 21, 2024
3c9df0b
add hierarchical resolve
eelcovdw Feb 21, 2024
0400edc
implement syncing endpoints
koenvanderveen Feb 21, 2024
a5bb227
fix hierarchical repr
eelcovdw Feb 21, 2024
1de4bb8
merge
eelcovdw Feb 21, 2024
f5e8b14
add delete
eelcovdw Feb 22, 2024
844ff32
added test for sync flow
teo-milea Feb 22, 2024
63f37c4
fix bytestring repr
eelcovdw Feb 22, 2024
acc815d
add previous state to syncstate
eelcovdw Feb 22, 2024
bf7f632
cleanup
eelcovdw Feb 22, 2024
9a0bf38
fix api bug for non in memory nodes
teo-milea Feb 22, 2024
fc75d3b
refactor syncing function
koenvanderveen Feb 22, 2024
6306048
None check fix for ActionObjectEmpty
eelcovdw Feb 27, 2024
06d45a1
Merge branch 'node-state-sync' into eelco/sync-hierarchical-resolve
eelcovdw Feb 27, 2024
f861ca0
remove hierarchical notebook, update syncing notebook
eelcovdw Feb 27, 2024
ef4b8b4
Merge pull request #8519 from OpenMined/eelco/sync-hierarchical-resolve
eelcovdw Feb 27, 2024
c2da7fd
merge
koenvanderveen Feb 27, 2024
26bf05f
decouple output history
eelcovdw Feb 27, 2024
8e2598b
remove get_dependencies
eelcovdw Feb 27, 2024
cb63567
Merge branch 'node-state-sync' of github.com:OpenMined/PySyft into no…
eelcovdw Feb 27, 2024
b708474
fix
eelcovdw Feb 27, 2024
c5bf74b
fix get_output_policy, fix outputpolicy
eelcovdw Feb 28, 2024
7ad6b2f
refactor diff_state
teo-milea Feb 28, 2024
9f8b23a
cleanup
koenvanderveen Feb 28, 2024
8cb552e
merge
koenvanderveen Feb 28, 2024
370fa55
merge in remove nested request versiongin
koenvanderveen Feb 29, 2024
b3c4b28
decouple code status
eelcovdw Feb 29, 2024
076bc40
merge
eelcovdw Feb 29, 2024
b075bb3
decouple status wip
eelcovdw Feb 29, 2024
f213022
make decoupling work
koenvanderveen Feb 29, 2024
270ea81
reprs
koenvanderveen Feb 29, 2024
65042ce
fix repr for resolve diff
eelcovdw Mar 1, 2024
c6a1106
fix visual hierarchy
eelcovdw Mar 1, 2024
a68902a
fix visual hierarchy
eelcovdw Mar 1, 2024
f2e19b2
Merge pull request #8534 from OpenMined/eelco/decouple_output_history
eelcovdw Mar 1, 2024
6ba3c5a
fix read permission for private objects
koenvanderveen Mar 2, 2024
79bafb2
fix repr + order for tables
eelcovdw Mar 4, 2024
8dc2b37
fix tests
eelcovdw Mar 4, 2024
e125be1
merge dev
koenvanderveen Mar 4, 2024
f4c0bb4
fix tests
eelcovdw Mar 4, 2024
e95dc84
Merge pull request #8550 from OpenMined/eelco/node-state-sync-fix-tests
eelcovdw Mar 4, 2024
236233a
fix request repr
eelcovdw Mar 4, 2024
b5e9131
fix test
koenvanderveen Mar 4, 2024
9987ad9
fix test
koenvanderveen Mar 4, 2024
0ce7113
fix tests
eelcovdw Mar 4, 2024
7f49bfe
Merge branch 'node-state-sync' of github.com:OpenMined/PySyft into no…
eelcovdw Mar 4, 2024
c9347f4
fix test
koenvanderveen Mar 4, 2024
34aceff
Merge branch 'node-state-sync' of github.com:OpenMined/PySyft into no…
koenvanderveen Mar 4, 2024
f8d0525
Merge branch 'dev' of github.com:OpenMined/PySyft into node-state-sync
koenvanderveen Mar 4, 2024
08b9df4
fix mypy
eelcovdw Mar 4, 2024
29e2701
linting
koenvanderveen Mar 4, 2024
0fff05d
Merge branch 'node-state-sync' of github.com:OpenMined/PySyft into no…
koenvanderveen Mar 4, 2024
0125ff3
fix sync service instrument
eelcovdw Mar 4, 2024
0110aaf
Merge branch 'node-state-sync' of github.com:OpenMined/PySyft into no…
eelcovdw Mar 4, 2024
c140c81
fix tests
koenvanderveen Mar 5, 2024
5eb2291
Merge branch 'node-state-sync' of github.com:OpenMined/PySyft into no…
koenvanderveen Mar 5, 2024
1c8704f
Merge branch 'dev' of github.com:OpenMined/PySyft into node-state-sync
koenvanderveen Mar 5, 2024
82436d2
fix notebook test
koenvanderveen Mar 5, 2024
df54b09
remove notebooks
koenvanderveen Mar 5, 2024
6deb2de
fix notebook
koenvanderveen Mar 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions notebooks/api/0.8/03-data-scientist-download-result.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
},
"outputs": [],
"source": [
"node = sy.orchestra.launch(name=\"test-domain-1\", port=\"auto\", dev_mode=True)"
"node = sy.orchestra.launch(name=\"test-domain-1\", dev_mode=True)"
]
},
{
Expand Down Expand Up @@ -166,19 +166,19 @@
]
},
{
"cell_type": "markdown",
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"Because the output policy is `OutputPolicyExecuteOnce`, this function cannot be run with other inputs. We can verify the validatiy of the policy as follows"
"domain_client.code"
]
},
{
"cell_type": "code",
"execution_count": null,
"cell_type": "markdown",
"metadata": {},
"outputs": [],
"source": [
"ops.valid"
"Because the output policy is `OutputPolicyExecuteOnce`, this function cannot be run with other inputs. We can verify the validatiy of the policy as follows"
]
},
{
Expand All @@ -189,8 +189,8 @@
},
"outputs": [],
"source": [
"assert isinstance(ops.valid, sy.SyftError)\n",
"assert ops.count == 1"
"assert isinstance(ops.is_valid, sy.SyftError)\n",
"assert ops.count > 0"
]
},
{
Expand Down Expand Up @@ -231,7 +231,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.2"
"version": "3.9.16"
},
"toc": {
"base_numbering": 1,
Expand Down
4 changes: 2 additions & 2 deletions notebooks/api/0.8/05-custom-policy.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
},
"outputs": [],
"source": [
"node = sy.orchestra.launch(name=\"test-domain-1\", port=\"auto\", dev_mode=True)"
"node = sy.orchestra.launch(name=\"test-domain-1\", port=\"auto\", dev_mode=True, reset=True)"
]
},
{
Expand Down Expand Up @@ -412,7 +412,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
"version": "3.9.16"
},
"toc": {
"base_numbering": 1,
Expand Down
2 changes: 1 addition & 1 deletion notebooks/tutorials/data-scientist/05-syft-functions.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -641,7 +641,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
"version": "3.9.16"
},
"toc": {
"base_numbering": 1,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -682,6 +682,14 @@
"source": [
"assert result_ptr.syft_action_data == 813"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d632521",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand Down
2 changes: 1 addition & 1 deletion notebooks/tutorials/model-auditing/colab/01-user-log.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -687,7 +687,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
"version": "3.9.16"
},
"toc": {
"base_numbering": 1,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,20 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
"version": "3.9.16"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -558,7 +558,20 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
"version": "3.9.16"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@
"outputs": [],
"source": [
"# gettting a reference to the user code object\n",
"user_code = change.link\n",
"user_code = change.code\n",
"\n",
"# viewing the actual code submitted for request\n",
"user_code.show_code"
Expand Down Expand Up @@ -304,7 +304,20 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
"version": "3.9.16"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,20 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
"version": "3.9.16"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
Expand Down
22 changes: 20 additions & 2 deletions notebooks/tutorials/pandas-cookbook/01-reading-from-a-csv.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -699,6 +699,16 @@
"request = project_notification.link.events[0].request"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6508050f",
"metadata": {},
"outputs": [],
"source": [
"func = request.code"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -708,8 +718,8 @@
},
"outputs": [],
"source": [
"func = request.changes[0].link\n",
"op = func.output_policy_type"
"# func = request.code\n",
"#"
]
},
{
Expand Down Expand Up @@ -853,6 +863,14 @@
"source": [
"node.land()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e80dab85",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -889,8 +889,7 @@
"outputs": [],
"source": [
"request = project_notification.link.events[0].request\n",
"func = request.changes[0].link\n",
"op = func.output_policy_type"
"func = request.code"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1003,8 +1003,7 @@
"outputs": [],
"source": [
"request = project_notification.link.events[0].request\n",
"func = request.changes[0].link\n",
"op = func.output_policy_type"
"func = request.code"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -775,8 +775,7 @@
"outputs": [],
"source": [
"request = project_notification.link.events[0].request\n",
"func = request.changes[0].link\n",
"op = func.output_policy_type"
"func = request.code"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -962,8 +962,7 @@
"outputs": [],
"source": [
"request = project_notification.link.events[0].request\n",
"func = request.changes[0].link\n",
"op = func.output_policy_type"
"func = request.code"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -865,8 +865,7 @@
"outputs": [],
"source": [
"request = project_notification.link.events[0].request\n",
"func = request.changes[0].link\n",
"op = func.output_policy_type"
"func = request.code"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -920,8 +920,7 @@
"outputs": [],
"source": [
"request = project_notification.link.events[0].request\n",
"func = request.changes[0].link\n",
"op = func.output_policy_type"
"func = request.code"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -871,8 +871,7 @@
"outputs": [],
"source": [
"request = project_notification.link.events[0].request\n",
"func = request.changes[0].link\n",
"op = func.output_policy_type"
"func = request.code"
]
},
{
Expand Down
9 changes: 9 additions & 0 deletions packages/syft/src/syft/client/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@

if TYPE_CHECKING:
# relative
from ..node import Node
from ..service.job.job_stash import Job


Expand Down Expand Up @@ -958,6 +959,14 @@ def from_change_context(cls, context: ChangeContext) -> NodeIdentity:
verify_key=context.node.signing_key.verify_key,
)

@classmethod
def from_node(cls, node: Node) -> NodeIdentity:
return cls(
node_name=node.name,
node_id=node.id,
verify_key=node.signing_key.verify_key,
)

def __eq__(self, other: Any) -> bool:
if not isinstance(other, NodeIdentity):
return False
Expand Down
18 changes: 17 additions & 1 deletion packages/syft/src/syft/client/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -531,6 +531,7 @@ def create_project(
def sync_code_from_request(self, request):
# relative
from ..service.code.user_code import UserCode
from ..service.code.user_code import UserCodeStatusCollection
from ..store.linked_obj import LinkedObject

code: Union[UserCode, SyftError] = request.code
Expand All @@ -543,7 +544,7 @@ def sync_code_from_request(self, request):

def get_nested_codes(code: UserCode):
result = []
for __, (linked_code_obj, _) in code.nested_codes.items():
for _, (linked_code_obj, _) in code.nested_codes.items():
nested_code = linked_code_obj.resolve
nested_code = deepcopy(nested_code)
nested_code.node_uid = code.node_uid
Expand All @@ -558,12 +559,27 @@ def get_nested_codes(code: UserCode):
code.nested_codes = updated_code_links
return result

def get_code_statusses(codes: List[UserCode]) -> List[UserCodeStatusCollection]:
statusses = []
for code in codes:
status = deepcopy(code.status)
statusses.append(status)
code.status_link = LinkedObject.from_obj(status, node_uid=code.node_uid)
return statusses

nested_codes = get_nested_codes(code)
statusses = get_code_statusses(nested_codes + [code])

for c in nested_codes + [code]:
res = self.code.submit(c)
if isinstance(res, SyftError):
return res

for status in statusses:
res = self.api.services.code_status.create(status)
if isinstance(res, SyftError):
return res

self._fetch_api(self.credentials)
return SyftSuccess(message="User Code Submitted")

Expand Down
Loading
Loading