BlobFile List
DockerWorker List
Id: a64e3e9d68984e8f93f24e55a5f1d195
\n", + "Request time: 2023-11-10 16:17:59
\n", + " \n", + " \n", + "Changes: Request to change main_function to permission RequestStatus.APPROVED.
\n", + "Status: RequestStatus.PENDING
\n", + "Requested on: Quizzical_pearl of type Domain owned by info@openmined.org
\n", + "Requested by: Jane Doe (info@openmined.org)
\n", + "DockerWorker List
Job List
Job List
Id: 56f76f67cf544edb8894139a92677a83
\n", + "Request time: 2023-11-24 15:37:14
\n", + " \n", + " \n", + "Status: RequestStatus.PENDING
\n", + "Requested on: Test-domain-helm2 of type Domain
\n", + "Requested by: Jane Doe (info@openmined.org)
\n", + "Changes: Request to change process_all to permission RequestStatus.APPROVED. Nested Requests not resolved.
\n", + "Id: 56f76f67cf544edb8894139a92677a83
\n", + "Request time: 2023-11-24 15:37:14
\n", + " \n", + " \n", + "Status: RequestStatus.PENDING
\n", + "Requested on: Test-domain-helm2 of type Domain
\n", + "Requested by: Jane Doe (info@openmined.org)
\n", + "Changes: Request to change process_all to permission RequestStatus.APPROVED.
This change requests the following nested functions calls:
├──middle_job
├────middle_middle_job
├──────process_batch
.
UserCode List
Job List
Would the Unix Philosophy when applied to country building help with this problem?\n======\nsova\nIf all legislation followed the model presented by git (versioning,\nincrements, branches, merges, and total transparency) I think that it would\nreflect positively on a true democracy.\n\nThe next step, however, would be to educate the populous so that all voters\nwere informed, and that voters would be presented (in an elegant fashion) with\nwhat is relevant to their districts on the three tiers of national, state, and\nlocal policy. I don't know if Unix has a good metaphor or reflection of this,\nbut unix is meant to be a) modular and b) minimalist, so if we can sponsor the\nidea of true modularity in voting, I think we could see some full-\nparticipation schemes that are not overwhelming. I don't have to vote on every\nissue, but could vote on collections of issues that reflect my general\nideology or current understanding of what best suits the republic.\n\nAnother issue though, is ownership. In the Feudalistic Republic of the United\nStates (as of 2016) it's hard to describe a system that could be adopted\nreasonably that promotes the idea that all the nation belongs to everyone in\nit. We have some things like \"the right to life, liberty, and property [often\nmisquoted as 'happiness' at the end here]\" and how does one reconcile this\nidea of property with a truly harmonious community? Good question.\n\nSo in short, the basis of the Unix philosophy would help (especially with law\nversioning, that is just what needs to happen and is so brilliant and clear I\nam surprised there is not greater traction for it). All Laws need time limits\n(and easy renew options if they are good)... And the entire populous needs\nhigher quality information that [forces?] causes people to consider the\ncommunity at large.\n\n\/rattle like a snake\n\n~~~\noftenwrong\nI have been quietly advocating for version controlled legislation for a long\ntime. Here in Massachusetts, bills describe what they would change in the text\nof a statute directly - a bit like an ed script. Here's an actual example:\n\n>SECTION 2. Said section 35 of said chapter 123, as so appearing, is hereby\nfurther amended by striking out the words \u201cis an alcoholic or substance\nabuser\u201d, in lines 17 and 18, and inserting in place thereof the following\nwords:- has an alcohol or substance use disorder.\n\n>SECTION 3. Said section 35 of said chapter 123, as so appearing, is hereby\nfurther amended by inserting after the word \u201ca\u201d, in line 36, the third time it\nappears, the following word:- qualified.\n\n>SECTION 4. Said section 35 of said chapter 123, as so appearing, is hereby\nfurther amended by striking out the fourth and fifth paragraphs and inserting\nin place thereof the following 3 paragraphs:-\n\nAs someone who attempts to keep informed about changes to the law, this style\nis a huge obstacle. It often necessitates a lot of manual piecing-together in\norder to form a complete view of the final change. A simple diff view would\nmake it much easier to understand.\n\nI have considered tracking changes to the law in git, including representing\nbills as branches, as a side project idea, but I determined it would require\nfar more effort than I am willing to put in.\n\n~~~\nsova\nWow sir. That is really a great list of examples! It actually seems [very\nvery] feasible to make a simple system that could automate this based on the\nlanguage used. It would be a worthwhile endeavor, but like you say, would take\na lot of time\/effort investment.\n\nPerhaps an open-source effort that does this (tracks and updates current laws\nand shows diffs) could be a worthwhile beginning?\n\nI think every senator and representative that has ever had to amend\nlegislation would delight at the thought.\n\n~~~\noftenwrong\nI was only considering doing it manually. I don't know how feasible it would\nbe to automate the conversion process. The formatting and language used in\nthese \"edit script\"-style bills varies considerably, as they are written for\nhumans by humans with no standardisation.\n\n~~~\nsova\nHonestly, this seems like one of the more realistic problems NLP could\nactually solve. Yes there may be many variants, say 100 or even 1000 different\nstructures and vocabularies for updating versions, but a differential neural\nnetwork where you have inputs (like the pre-amended law) and outputs (like the\nlaws after the \"amendments\" or version bumps) would actually be perfect for\nlearning what means what and when to do it.\n\nIt would be the perfect grad project for someone interesting in bridging the\ngap between computation\/machine learning and legislation.\n\nOf course, it would be a little tedious setting up the learning (thousands of\nsets of input cases and output cases) but in the end the findings could be\nused across the board.\n\n------\narkitaip\nHow do you define The Unix Philosophy as applied to a country? Software\ndoesn't come close to the complexity of an entire country so your analogy\ncould possibly be fundamentally mismatched...\n\n~~~\nNumberwang\nWell I believe that fundamentally the complexity of a country is to a large\nextent historical artifacts.\n\nHow do you think the relations between institutions would be different, how do\nyou think they would perform their functions differently? What would their\nstructures be?\n\nOr focus on some specific example -How would voting be different? -How would\nregistering for a licence be different? -How would taxation be different?\n\n~~~\nsova\nI think with taxation we could also do very cool things: Say your nation taxes\nat 30%, what if every voter had a subset of that value (say, 12.2%) that they\ncould choose which district or set-of-needs to fund?\n\nLike maybe I want my 12.2 to go to education for kids 6mnths-12years, or maybe\nI want to fund state medicines, and my neighbor and I both pay the base rate\nthat covers necessities like roads and stuff, but he may fund shelters instead\nof medicine specifically with his 12point2. It could be really wonderful.\n\nIn effect, people may become more participatory in their own governing\nsystems, and could actually direct funds instead of relying on bill-makers to\nfigure out where to spend monies\/resources.\n\n------\nangersock\nThe plumbing, presumably.\n\n~~~\nbbcbasic\nThe sewerage and garbage can be piped into \/dev\/null\n\n------\noftenwrong\nNational PKI. Every citizen would have a key pair. I believe I have read that\nEstonia has implemented this.\n\n~~~\nNumberwang\nWhat could it be used for?\n\n~~~\nsova\nVoting! And easily verifying a) your vote was\/is counted and b) is accurate\nfor what issues\/candidates you voted for. In fact, we could eliminate most\ncandidates because they are only there to \"represent\" the wills\/intentions of\ntheir constituents. Gloabl PKI pairings for voting would eliminate the need\nfor a lot of \"representatives\" and we could do more direct forms of democracy\ninstead!\n\n","meta":"{'id': '12585834'}"}
+{"text":"\nNumber of Users on Each Plan - danw\nhttp:\/\/www.barenakedapp.com\/dropsend\/number-of-users-on-each-plan\n======\njwecker\nnice post. In my experience also the lowest paying accounts are the most\ndifficult to maintain- out of proportion certainly to the revenue they bring\nin. However, one thing it didn't mention here is to remember not to discount\nmarket share. In lots of apps the higher subscription plans will only be\nupsales- no one will jump straight into the business account, for example. And\nin some cases your low paying accounts are doing a lot of evangelizing for\nyour product, or not using a competitors product, etc. Keep it balanced, for\nsure, but get lots of users.\n\n","meta":"{'id': '3714'}"}
+{"text":"Ask HN: What are well designed SaaS websites? - piu\n======\nksec\nStripe.com\n\nSimple, Consistent, fast , effective.\n\nThere are many other listed here as well. They mostly follows the same layout\nand pattern. What separate them are wording and graphics. Using simple words,\nand shortest sentence possible to describe your SaaS, and choice of graphics,\nwhich really is a matter of personal taste.\n\nI think Stripe manage to do this very well.\n\nOff Topic: Did Stripe ever talk about their Ruby Stack?\n\n~~~\nsimlevesque\nOn Stripe I really like that you can see the logs of every API call you've\never made with the request headers and body and response body... It makes\nworking with it much easier than Braintree.\n\n~~~\nMandatum\nDoes Stripe write about how they handle storing those requests\/responses?\nSeems like this could get very expensive, very quickly.\n\n------\nkenning\nI think turbotax has a pretty phenomenal interface if you're in the bracket of\npeople with really simple taxes. Two and three years ago, my taxes took me\nabout an hour.\n\nDepending on what you're looking for, you may also be interested in aping\ntheir freemium model, where the first time you use the service is free and\nsets you up quite well to reuse the service next year and pay $40 for one of\ntheir obnoxious services. As a customer it was quite frustrating but it\nsucceeded in getting me to pay $40 the second year, and had I not gone far out\nof my way to remove the \"plus\" and \"premium\" features I would have ended up\npaying ~$100 the first year and $140 total the second.\n\nThe third year I switched to a competitor and got to use their service for\nfree. In a way, using turbotax felt like a great UX mixed with a battle to\nread everything extremely carefully and retread my steps to avoid paying\nanything; to me, this is not all that morally reprehensible because it\nadversely affects people who don't value their money as much as their time.\nHowever it also seemed predatory in that a non-tech-savvy user such as my\nparents would likely be tricked into paying higher costs for essentially no\nadded value.\n\n~~~\ntootie\nThey have a really solid approach and keeping each step really straightforward\nand discrete to avoid overwhelming you with too much to think about at once.\nIt still fails really hard when you get to anything outside their flow. I had\nto spend time googling the awkward set of steps needed to deduct mortgage\ninterest. Ultimately, it wasn't hard, but it wasn't at all obvious how to do\nit.\n\n~~~\nbeamatronic\nUm... maybe you had a strange situation but usually for mortgage interest on\nyour home , your lender sends you a form with essentially 1 number on it and\nyou just enter this form into TurboTax when it asks you.\n\n------\nbjterry\nIt seems the question is ambiguous. Everyone is responding with the marketing\nwebsites of SaaS compnaies, but I interpreted it as asking for well-designed\ninternal interfaces of SaaS websites. Would love to see examples of that which\npeople think are particularly great. Personally I've always found Gusto and\nBasecamp to have very good interfaces. Stripe's internal interface (which\nothers have mentioned for their public site) gets the job done but I would\nhardly call it great.\n\n------\nphilip1209\nSome of my favorites:\n\n[https:\/\/mailchimp.com](https:\/\/mailchimp.com)\n\n[https:\/\/transitapp.com\/](https:\/\/transitapp.com\/)\n\n[https:\/\/www.intercom.com\/](https:\/\/www.intercom.com\/)\n\n[https:\/\/lattice.com\/](https:\/\/lattice.com\/)\n\nI'm fond of what we have built:\n[https:\/\/www.moonlightwork.com](https:\/\/www.moonlightwork.com)\n\n~~~\nwhitepoplar\nHey, curious about your experience with Mailchimp. I've noticed that people\nseem to either love it or hate it. What do you think they do well? Where do\nthey fall short? (if at all)\n\n~~~\njonathan-kosgei\nI hate mailchimp and prefer to use tinyletter.com. I can't talk enough about\nhow much I love tinyletter!\n\n~~~\nflaviocopes\nTinyLetter is amazing. Simple, easy to use, just does what you need without\ntemplates, campaigns and other stuff that gets in the way between you and\nsubscribers receiving an update from you.\n\n------\nanacleto\nNeedless to say Stripe.com \\-\n[https:\/\/www.plainflow.com\/](https:\/\/www.plainflow.com\/) \\-\n[https:\/\/sentry.io](https:\/\/sentry.io) \\-\n[https:\/\/slack.com](https:\/\/slack.com) \\-\n[https:\/\/figma.com](https:\/\/figma.com) \\-\n[https:\/\/basecamp.com](https:\/\/basecamp.com)\n\n~~~\nphilfrasty\nI found Slack rather poor in explaining what they are doing. This text is\nbasically their entire landing page.\n\n\"When your team needs to kick off a project, hire a new employee, deploy some\ncode, review a sales contract, finalize next year's budget, measure an A\/B\ntest, plan your next office opening, and more, Slack has you covered.\"\n\nDo they offer A\/B testing? HR tools? Code deployment? Who would have guessed\nit is chat.\n\nTheir \/features page does a better job: \"It simplifies communication. Slack\nbrings all your team's communication together, giving everyone a shared\nworkspace where conversations are organized and accessible.\"\n\n~~~\nanacleto\nTrue.\n\nBut that's actually a common trend. When the company's brand gets bigger and\nstronger in people's mind, company position slowly switches from\n\n1 Product attributes\n\n2\\. Product benefits\n\n3\\. Emotional benefits\n\n4\\. Something bigger\n\nThis applies well to every type of product. SaaS included.\n\nThis is a great essay on the topic:\n[https:\/\/medium.com\/speroventures\/branding-for-\nbuilders-19e10...](https:\/\/medium.com\/speroventures\/branding-for-\nbuilders-19e103ef3f1d)\n\n------\nlwansbrough\nI was wondering the same thing the other day: looking for inspiration but also\nexperienced recommendations and UI patterns. Found this with a quick Google (I\nhave no affiliation): [https:\/\/blog.chartmogul.com\/saas-landing-\npages](https:\/\/blog.chartmogul.com\/saas-landing-pages)\n\nAlso I found Pinterest to be a good resource for finding designs (more so than\nDribbble, Behance, etc. surprisingly.)\n\n------\nspking\n[https:\/\/baremetrics.com](https:\/\/baremetrics.com)\n\n[https:\/\/sendgrid.com](https:\/\/sendgrid.com)\n\n[https:\/\/www.drift.com](https:\/\/www.drift.com)\n\n[https:\/\/lookback.io](https:\/\/lookback.io)\n\n[https:\/\/reply.io](https:\/\/reply.io)\n\n~~~\nbriandear\nBaremetrics for sure. Really effective \u2014 the dashboard gives you all the\nimportant data quickly and then you can easily drill down. I use their product\nseveral times a day and it\u2019s the best interface of all of the many services I\nuse.\n\n------\njonathanbull\n[https:\/\/lookatthatsaas.com](https:\/\/lookatthatsaas.com) is a good resource\nfor inspiration.\n\n------\nqstearns\n[https:\/\/segment.com\/](https:\/\/segment.com\/) seems to take design really\nseriously. They also have a pretty nice React toolkit here:\n[https:\/\/segmentio.github.io\/evergreen\/?selectedKind=alert&se...](https:\/\/segmentio.github.io\/evergreen\/?selectedKind=alert&selectedStory=Alert&full=0&down=0&left=1&panelRight=0&downPanel=storybook%2Factions%2Factions-\npanel)\n\n------\ntschellenbach\nI frequently compare [https:\/\/getstream.io\/](https:\/\/getstream.io\/) with\n[http:\/\/stripe.com\/](http:\/\/stripe.com\/),\n[https:\/\/www.mapbox.com\/](https:\/\/www.mapbox.com\/), sendbird.com, algolia.com,\npusher.com and [https:\/\/layer.com\/](https:\/\/layer.com\/)\n\n------\nruairidhwm\n[https:\/\/stripe.com](https:\/\/stripe.com) \\- It's beautiful but conveys all the\ninformation that you need quickly. It also has excellent copy.\n\n[https:\/\/canny.io](https:\/\/canny.io) \\- Very crisp design and it conveys the\nuse case really well.\n\n[https:\/\/baremetrics.com](https:\/\/baremetrics.com) \\- This has come such a\nlong way and has stunning design.\n\n------\nigorv\nNot really a SaaS website, but I really dig this\n[https:\/\/district0x.io\/](https:\/\/district0x.io\/)\n\n~~~\n2bitencryption\nI'm not too fond of that \"stay up to date\" modal, which tries to mimic system\nnative UI.\n\n------\nwhitepoplar\n[https:\/\/dnsimple.com](https:\/\/dnsimple.com)\n\n[https:\/\/basecamp.com](https:\/\/basecamp.com)\n\n[https:\/\/sentry.io](https:\/\/sentry.io)\n\n[https:\/\/semaphoreci.com](https:\/\/semaphoreci.com)\n\n[https:\/\/instapaper.com](https:\/\/instapaper.com)\n\nOld Heroku :-(\n\n~~~\njohnhenry\nWhat's changed about Heroku that's made you unhappy? (Not an employee, just\ncurious).\n\n~~~\nwhitepoplar\nTake this copywriting, for example:\n\n2011: \"Forget Servers - Get up and running in minutes, and deploy instantly\nwith git. Focus 100% on your code, and never think about servers, instances,\nor VMs again.\"\n\n2018: \"Deploy and run apps on today's most innovative Platform as a Service -\nHeroku is a cloud platform based on a managed container system, with\nintegrated data services and a powerful ecosystem, for deploying and running\nmodern apps. The Heroku developer experience is an app-centric approach for\nsoftware delivery, integrated with today\u2019s most popular developer tools and\nworkflows.\"\n\nWhich is better?\n\n~~~\nHeyLaughingBoy\n2018: it describes what they do in a much clearer way.\n\n------\nsimantel\nFor lots of examples, check out\n[https:\/\/www.pages.xyz\/](https:\/\/www.pages.xyz\/)\n\n------\nCommanderData\nI recently came across toggl for time tracking and reporting.\n\nToggl - Time tracking -\n[https:\/\/toggl.com\/pricing\/](https:\/\/toggl.com\/pricing\/)\n\nTheir pricing page is one of a nicest I've seen, really easy to grasp but also\nfunctional eye candy.\n\nI even hoped it was a WP template so I could customize one myself.\n\n------\nleonroy\nClubhouse is an excellent example of site and web app and their approach and\nintegration with both has clearly had a LOT of thought put into it:\n[https:\/\/clubhouse.io](https:\/\/clubhouse.io)\n\nProbably use it more than any other SaaS and am glad it\u2019s so good.\n\n------\ndeadcoder0904\nFind some great Inspiration at [https:\/\/hyperpixel.io](https:\/\/hyperpixel.io)\n\n------\noferzelig\n[https:\/\/omnystudio.com\/](https:\/\/omnystudio.com\/)\n\n------\ncyberferret\nCan I chime in here, not with a link to any specific site, but just as a call\nout to patterns that I am seeing recently.\n\nA lot of sites now have lists and content that updates automatically as things\nhappen on the back end. One good example is Intercom. I have their screen open\n24x7 on the first tab of my browser so I can monitor users on our site. I love\nhow it updates the 'time last seen' dynamically, and I usually have my\ncustomer list sorted by the 'time last seen' field.\n\nBut sometimes, while the content of the list fields are updated in real time,\nthe sorting of the list is not, and the list goes out of order (i.e. customers\nwho re-login recently are still shown lower down in the list that customers\nwho logged in an hour ago even though the 'last login time' is more recent.\n\nI wish there was a way in these instances to just refresh the list within the\npage, without doing an entire browser page refresh, which could take up to 10\nseconds in the old Intercom UX. Also, while talking about Intercom, jumping\nbetween the Customers page and the Conversations page could also take anything\nfrom 5 to 10 seconds on my browser, and there was NO indication that anything\nwas happening in the meantime, which increased confusion and frustration. I\nthink we need to bring back the hourglass or some other 'waiting' indicator\nfor transitions that take a while.\n\n(NB: The new Intercom UX has improved on the waiting delay significantly, but\nnot the sort ordering of the customer list).\n\nSomeone also mentioned the Stripe design (of their back end, not their\nmarketing site). I tend to like the new design of their admin panel, however\ntheir menu hierarchy was a little confusing, making it hard to find things a\nlot of the time. Also, the redesign tends to break the 'back button' behaviour\na lot. I tend to spend a lot of my time on the Stripe admin panel looking at\nwebhook logs etc., and every time I bring up the log listing, then drill down\nto an entry I can't seem to go 'back' to the list easily without the system\nrebuilding the entire list each time. Makes it frustratingly slow to try and\nfind the exact log entry I want when I have to spend so much time waiting for\npage refreshes.\n\nIn summary, I think we need to go back to these 'old fashioned' design\nconstructs which aren't considered \"trendy\" any more:\n\n* Give the user some sort of 'waiting' indicator if a page redraw is going to take time.\n\n* If a list on your page refreshes in the background, and your user can sort the list, make sure you update the sort order as well as the content\n\n* Don't break the back button behaviour if you can help it.\n\n------\njacobwg\n[https:\/\/webflow.com\/](https:\/\/webflow.com\/) \\- complex product, but the\nmarketing site makes it clear and understandable\n\n------\nhartator\nShameless plug, I kind of like the work we did on our own SaaS website:\n[https:\/\/serpapi.com](https:\/\/serpapi.com)\n\n------\nvelp\npitchbook.com\n\nInteresting mix of content and product information. I like how it's laid out\nas well\n\n------\nMojah\nMy favorites:\n\n\\- stripe.com\n\n\\- ohdearapp.com\n\nSimple, to the point & clean layouts.\n\n------\njohnhenry\nDid you mean that the services themselves are well designed or are you\nreferring to pages that describe them?\n\n------\njiveturkey\ndoes ecommerce count? mcmaster.com\n\n~~~\nmanuaero\nagreed ... mcmaster is one of the best designed sites.\n\n------\nfairpx\nChiming in after seeing us get mentioned here (context: lead designer @\n[http:\/\/Fairpixels.pro](http:\/\/Fairpixels.pro))\n\nWorking with engineers of b2b saas companies every day, for more than a year\nand having analysed all the best SaaS companies who have 10+ internal\ndesigners, I found a couple of principles that anyone can apply to make their\nwebsite look decent:\n\n* Consistency - One practical example: If you use a 4px border-radius, use a 4px radius everywhere. It may sound small, but having a consistent experience across your application makes the product feel so much more polished to your users. Don't use multiple fonts, different navigation menus etc. Keep it consistent.\n\n* Reduction - If anything, design isn't about adding more things to make it 'look nice'. Try to remove as many things as you can. If something doesn't serve a specific function, then remove it. Every pixel should make sense. The more you put in front of your users, the more brain power it'll require to process.\n\n* Divide - This is mostly UX, but one thing I see so many get wrong. A lot of SaaS apps overwhelm their users. They present them with all the features upfront. Whether it's a long onboarding form, or a dashboard with 50 actions one could take. By splitting up things in different ways, you can guide the user through the experience. Your signup process for example (that might be a big block in conversion) might be made so much better if you ask for certain types of information later on in the process.\n\n~~~\nvincentmarle\nI very much like your fixed fee \/ month business model. Exactly what I need, I\nwill likely become a customer soon.\n\nIs there a similar service out there that has fixed pricing for web\/app\ndevelopment?\n\n~~~\nredmaple\nsaw this few weeks ago:\n[https:\/\/greenpine.co\/#pricing](https:\/\/greenpine.co\/#pricing)\n\n------\niampaul\nMost SaaS businesses are run by engineers and unfortunately many of them\/us\nlack the eye for style. That said, here are two of my favorites:\n\n[http:\/\/fairpixels.pro](http:\/\/fairpixels.pro) \\- I found these guys here on\nHN and their work seems spot on.\n\n[https:\/\/www.spotify.com\/](https:\/\/www.spotify.com\/) \\- their simple design\nand IPO should be an example for fellow engineers who\u2019re building saas.\n\n~~~\nrahimnathwani\nFairpixels doesn't appear to be a SaaS service, but a service company that\ndoes design (not only for SaaS).\n\nI'm curious:\n\n\\- is there a particular SaaS designed by fairpixels that you consider an\nexample of good SaaS design?\n\n\\- do you have any relationship with Fairpixels? Your HN account has posted 2\ncomments since being created, and both those comments recommend fairpixels.\n\n~~~\niampaul\nIve been following their progress for over a year and am a customer. They\u2019ve\nstructured their website and business like a Saas. I don\u2019t know about all of\ntheir customers but I love the work they did for Uphex.com for example.\n\n------\nsoulchild37\nWould recommend [https:\/\/stunning.co\/](https:\/\/stunning.co\/) , I like the\nfloating tube animation and the increasing recovered amount is really\nappealling.\n\n~~~\nRjevski\nMeh, I disagree, the design looks dated.\n\n~~~\nsamstave\nI agree - it feels poorly-designed-2012\n\nBut thats not a knock on their offering - if their customers are happy and\nthey are doing a good job, then more power to their servers.\n\n","meta":"{'id': '16837683'}"}
+{"text":"\nYou think you know what teachers do. Right? Wrong. - mathattack\nhttp:\/\/www.washingtonpost.com\/blogs\/answer-sheet\/wp\/2014\/02\/22\/you-think-you-know-what-teachers-do-right-wrong\/\n======\nsramsay\nThis is true of being a professor as well (I certainly didn't understand what\nteaching really was about until I starting doing it).\n\nI've always thought that our graduate students should be made to take acting\nlessons, because there's an element of second-order persuasion you have to do\nin a classroom that's hard to learn and difficult to describe but that shares\nsome similarity to acting -- or maybe just rhetoric in the very ancient sense.\n\nYou can't just purvey information and mumble something about its importance.\nUltimately, you're modeling what it means to be an intellectual -- trying to\ngive your students certain habits of mind by showing them how those habits\nplay out in practice.\n\nWe also spend an enormous amount of time trying to devise strategies for\ndealing with students who just don't get it (and you quickly learn -- or\nbetter learn -- that this might be the most important part of the job).\n\nI could say more, of course. It's a very subtle set of skills -- more art than\nscience, as they say. It's hard to do it at the college level, and I think\nit's far, far harder to do it at the elementary level, where the stakes are\nmuch higher.\n\n~~~\nbarry-cotter\nWhat kind of third level institution do you work at? One is under the impress\nthat going from passable to outstanding in teaching has much, much less effect\non one's chances of getting tenure than going from mediocre to good in your\nresearch.\n\n~~~\numanwizard\nThey never said it was particularly important for career advancement. How did\nyou read that into their post?\n\nAlso, what's with the condescending sneery tone?\n\n~~~\nadestefan\nBecause every time a post on education comes on HN everyone thinks they know\nall the answers. The comments end up turning into a \"Well this is the real\nreason...\" or \"Everyone needs to be just like...\"\n\nThe discussions end up being so worthless that I now flag every education\nrelated post on HN because it's just not worth the time here.\n\n~~~\njedmeyers\nWhy do you think it's called master of Arts in teaching, and not master of\nScience?\n\n------\nSniperfish\nMy wife is a teacher. I am consistently shocked how much work she does in\nevenings, weekends. I earn more than twice as much as her and more than her\nmaximum salary cap (we are both early in our careers). She blows me away in\nher dedication and effort, it's a great inspiration for me to continually\nstudy and work harder.\n\nI mention it to people and always hear 'well maybe she is different but I've\nseen lots of teachers and they just do it for the holiday'. As if everyone is\nequally dedicated in any profession. As if the guy that sits at his computer\n'working' for hours a day is a more efficient or effective worker just because\nhe does more hours. As if outside observers of any industry can really spot\nwho is producing vs who is not.\n\n~~~\nmontecarl\nI can echo your story exactly. My wife teaches and is involved in an after\nschool program. It isn't football but has a similar time commitment. During\ncertain parts of the year she works 6 days a week often leaving the house at 8\nam and returning at 9 or 10 pm. It is insane. Two other teachers in her\ndepartment work similar hours. The pay per hour isn't very good once you\nfactor all of that in.\n\n~~~\nGotAnyMegadeth\nAt the other end of the spectrum, one of the teachers at my old school used to\nturn up at 8:30 and leave at 15:30. She used to put a video on, and then hand\nout worksheets to fill in whilst she marked the worksheets from the class\nbefore. Terrible teacher, luckily I only had her for a few weeks.\n\n~~~\nnumo16\nI have a few friends that are teachers and most of their schools wouldn't\nallow for this sort of thing to happen. Teachers aren't allowed to sit at\ntheir desk while a class is in session, they must be instructing or walking\naround (during a test, video, etc...). They get a planning period, where they\nmight have a chance to do some grading or lesson planning, if they don't need\nto meet with a parent or something. This means they need to either stick\naround school several hours after it lets out to grade work and do lesson\nplans, or bring it home and work on it that night.\n\n------\nsaosebastiao\n>The problem with teaching as a profession is that every single adult citizen\nof this country thinks that they know what teachers do. And they don't. So\nthey prescribe solutions, and they develop public policy, and they\neditorialize, and they politicize. And they don't listen to those who do know.\nThose who could teach. The teachers.\n\nSorry, I cant take this seriously. The teachers unions are one of the most\npolitically powerful entities in the US. They can make a candidate, and they\ncan break a candidate. They can pass and tank ballot measures...even ones\ncompletely unrelated to their jobs. They can protect drunkards and criminals\nfrom getting prosecuted, let alone fired. They are fine forcing their agenda\ndown our throats, but they cant take a little pushback?\n\n~~~\npbhjpbhj\n> _They are fine forcing their agenda down our throats_ \/\/\n\nThe agenda of ensuring children have access to life-enhancing educational\nopportunities?\n\n> _They can make a candidate, and they can break a candidate._ \/\/\n\nYou mean a political candidate? You really think that the combined voice of a\ngroup of teachers can do that against the weight of media conglomerates, other\nunions, rich lobbyists and other political groups? Any examples?\n\nPresumably under your assertion the education system in the USA is the one\nthat the teaching unions have won by political action and the politicos and\nbusiness people are looking on powerless to influence it?\n\n~~~\nPaulHoule\nWell, I can say that in two weeks of homeschooling I got my son to write more\nthan they did in five years.\n\nHe was having trouble with bullies and the school did nothing about it. They\npretty much gave up on teaching spelling completely. We found out that our\nschool is a \"magnet school\" for behaviorally disturbed \"special\" kids from\nother districts so kids in the rich school and kids in the poor school where\ncommunities complain a lot get to enjoy a safer environment because the rural\nschool gets all the psychotic kids.\n\nI gave up on them when the superintendent gave a \"town hall\" where he told the\nmother of a \"special\" kid that he was a partner in his education and he told\nme I should just butt out because he was the expert and there's a new paradigm\nand homework is obsolete and because I don't have a phone number to call to\nget Albany breathing down his neck.\n\nF the teacher's unions.\n\n~~~\nking_jester\nThe problems you experienced go beyond teachers unions. Dumping \"problem\" kids\ninto one school is a recipe for disaster and communities are not served by\nthat kind of thing at all (except those that dumped off students, although I\nwould argue those communities aren't fixing their underlying problems).\nAdministrator heavy, top down approaches that override community and teacher\nautonomy are a bad thing in general, and the obsession with testing over\nstandard lessons and homework is a huge problem with the way the public\neducation system is run.\n\nUltimately teachers as a professional class deserve a union. We see in other\nplaces and countries that the unions do not serve as an impediment to a\nquality public education, so we have to ask ourselves what is really going on\nwith current systems and unions that make the situation so shitty (esp. in New\nYork state).\n\n~~~\nPaulHoule\nI'm not saying that teachers shouldn't have a union, but from my perspective\nit is part of the problem rather than the solution more often than not.\n\nFor instance, they opened a charter school in our district which seems to be\nan honest effort to provide a safe (bullying free) environment for the high\nschool and there have been two people associated with the union who have just\nbeen consistently hateful trying to shut it down.\n\n~~~\nking_jester\nThe charter school movement is one of those things that draws strong opinions.\nInitiatives to provide safe school environments are good, but privatized\ncharter schools have a lot of downside in terms of how a community, parents,\nand teachers can retain control over how education happens. In New York state\nin particular, there has been a strong effort to close public schools and open\nprivate charters, which in my opinion is the wrong way to fix problems with\npublic education. The disagreement over charters isn't just a union thing,\nalthough public educators would be upset to see the system they work for\ndismantled instead of repaired.\n\n------\nShivetya\npuff piece, if not pure propaganda bordering on hyperbole.\n\nPeople and students respect teachers as a whole, what they do not respect and\nI bet many in the profession do as well is the inability to remove those who\nare not good teachers.\n\nIt is not a position one walks into without many upon many stories about what\nyour really getting into. My Aunt retired from the trade, her aggravations in\norder that I remember are, Administrative people(usually political\nappointees), other teachers, and parents. There were a few others but mostly\nthe tripe coming down from non teachers within the system seemed to be what we\nheard of.\n\nThat and the personal money she spent to have supplies because it was more\nimportant to blow money on marble floors than supplied, or having someone's\nwife\/kid\/friend in some advisement position that did nothing but occupy space.\n\nGuess what, I can say the same of some other service professions, having a\nneighbor who does night shifts as a nurse and hearing the horror stories of\nwhat she puts up with is enough to let me know some jobs come with extra\nmental if not physical stress.\n\nI think in the end we are all more than willing to heap accolades on good\nteachers. Its a system where the kids aren't first that irritates\n\n------\nsteveplace\nTeacher worship can only go so far.\n\nBecause this post makes the claim that _all_ teachers should be looked up to.\n\nMy entire family consists of teachers. They know who the bad teachers are.\nYou've got Paulina Pensioner who just shows old VHS tapes as a history\ncirriculum. Or Carl the Coach that knows, just _knows_ there's only one way to\nsolve this pre-algebra problem.\n\nAnd some teachers work hard. They bust their ass and bring grading home and\nlesson plan on the weekends.\n\nBut they aren't the problem. There's a bad system that keeps bad teachers in\nat the expense of the good.\n\nSo they design tests and standards as a way to \"firewall\" these bad teachers\nin, to turn their poor performance into mediocre performance. And there's a\ncost, because it removes the creativity and initiative from the good teachers.\n\nI understand that the goal of the author is to criticize common core, but\nwhile the conclusion is sound (Core is garbage) the reasoning is not.\n\nAnd the new standards being developed? One of the main proponents is the\nCouncil of Chief State School officers. Many (probably most) came from the\nteaching profession. Who know what it's like to be a teacher.\n\nThe author gives us some feel-good patronization about how teachers have it so\nhard and we have no right to impose standards upon them. But these standards\nexist because we can't fire bad teachers.\n\n~~~\nrmrfrmrf\nI don't think Core is garbage at all. I think there's a deeply ingrained\nculture of anti-intellectualism in US culture that needs to be nuked out of\nthe school system, and I honestly couldn't care less what the collateral\ndamage is.\n\n~~~\nsteveplace\nHere's the thing.\n\nYou like it when there's wide, sweeping cirriculum on the Federal level...\nwhen you agree with it.\n\nBut what happens if there's enough political pressure (it is a midterm\nelection cycle) to add ID into the cirriculum? Or maybe they look at feel-good\nmath that is just teaching to the test [1]?\n\nAnd that's the issue. Centralized power is great when you agree with it, but\nterrible in the wrong hands.\n\n[1] [http:\/\/www.momdot.com\/common-core-is-making-me-\nstupider\/](http:\/\/www.momdot.com\/common-core-is-making-me-stupider\/)\n\n~~~\nrmrfrmrf\nI agree with your point. I suppose I'm fortunate enough to also agree with the\ngoals of Common Core as they are today.\n\nOnto that article, however:\n\n1\\. I never use an academic degree as an indicator of intellectual capacity. I\nfind that some people are so objective-driven that they zoom right past the\npoint and straight to rageville when they don't understand something.\n\n2\\. A simple Google search on front-end estimation would have helped this mom\nrealize that the example given on the sheet is incorrect. I will concede that\nan effective teacher would have realized that the example given is incorrect\nand would have corrected it.\n\n(In front end estimation, you round the leftmost digit, so the example should\nactually be 400 + 300, not 300 + 200). IMHO 700 is actually a decent estimate\nfor 645, so I don't think there's a problem with the math itself. It's not\nreally feel-good math, but I think some people take for granted that\nestimation is not an innate ability.\n\nNow, it becomes another discussion altogether when the teacher is so horrible\nthat they refuse to accept that the example is wrong. But, I don't think I've\nseen evidence of that, so I won't accuse anyone of anything.\n\nEDIT: I just read some of the comments in that article, and it looks like some\ndistricts teach front-end estimation with truncation rather than rounding, in\nwhich case 300 + 200 = 500 would be correct.\n\nHere are a few more things to note: the parents here _assume_ that estimation\nand rounding are the same thing. That in itself isn't true.\n\nMore importantly, though, look at the _goal_ of the estimation -- to see if\nthe _actual_ answer, 645, is reasonable. That's _not_ the same thing as asking\nif 500 is a reasonable estimate of 645. I think the point of this exercise is\nfor kids to say \"ok, if I add these two numbers together, I _expect_ to get a\n3-digit number somewhere in the ballpark of 500.\" That is to say: if I add 354\nand 291, I shouldn't expect to get 20000 or 7 or 81 or 9750. It's just a\nsimple way of checking your work using a quick, easy method that you can do in\nyour head. Again, I find the value in this -- adding \"common sense\" to the\ncurriculum is definitely something I can get behind, but I understand that\nparents who aren't used to \"common sense on paper\" will struggle.\n\n------\nmildtrepidation\nI hate writing like this. Even if most people don't know the thing you're\nreferring to, basically telling the entire browsing population of the internet\n\"we're all stupid and here's why\" immediately leaves a bad taste, particularly\nfor people -- you know, like _teachers_ \\-- who _do_ know what teachers do, or\npeople who didn't make the assumption being assumed in the first place (which\nsays a lot more about the author than anything else).\n\nPedantic? Maybe. But to me this is a really childish way to make a point that\ncould be better stated in a way that doesn't instantly, baselessly denigrate\nthe reader, particularly when you're writing for a publication that banks on\nits credibility and reputation.\n\n------\npatmcc\nI have tons of respect and sympathy for teachers, but the argument I often\nhear for raising their pay (\"they work really hard, they're super important,\nit's a difficult job\") misses the central point.\n\nIt seems like we have enough teachers at the wages we currently pay. Teachers\nare willing to go into the profession despite the low wages, probably because\nthey want a satisfying job with good benefits. If we didn't have enough\nteachers...we'd have to raise wages. Supply and demand.\n\n~~~\nNursie\nAnd like many other situations which can be summed up as supply and demand, a\nrace to the bottom is an obvious outcome.\n\nMaybe we'd get better teachers if we paid more?\n\n~~~\npatmcc\nWe'd get better teachers if we paid more to good teachers. The problem is no\none can seem to agree on how to measure what makes a good teacher - one side\nis busy arguing seniority should be the primary measure, the other side argues\ntest scores, and neither one seems to want to spend any time or money figuring\nout an actually successful way to measure teacher skill.\n\n~~~\nmindslight\nYou could _ask the kids_. They certainly know which classes are engaging, and\nwhich are time-biding garbage. And ultimately, assuming a teacher isn't\nrunning a movie theatre, student interest _is_ the most important metric.\nYou'd of course have to keep the actual weighting process a bit fluid to avoid\nthe inmates gaining control of the asylum, but it should be quite\nstraightforward to pick out the extreme bad and extreme good teachers.\n\nIt would also be a good introduction to the rationale behind secret ballots,\nand when it is actually appropriate to lie.\n\n~~~\nameister14\nLook at ratemyprofessor and see how well an incredibly difficult professor\nthat is also engaging and interesting does; now imagine that in a situation\nwhere the people in his\/her class are forced to be there.\n\n~~~\nmindslight\nI took a quick look through that site, paging through my alma mater of a\ndecade ago. I do see pathologies in the ratings\/comments that remind me of\ncomplaints I would hear about professors from fellow students that were\nstressed, not getting the material, or used to a more structured environment.\nAnd if these ratings held weight with the university, I can definitely see\nprofessors dumbing down their lessons to avoid bad reviews. So I do see what\nyou're getting at with it going terribly wrong.\n\nStill, I think there's several key differences:\n\n1\\. Every school student would be rating their teachers, rather than just\nthose that loved a professor, had an axe to grind, or were encouraged to by an\nentertaining personality.\n\n2\\. The context would be \"closed\", with each teacher relative to their school,\nrather than open cross-institution competition with a front page of featured\n\"rockstar\" professors that make the rest seem inadequate.\n\n3\\. The high schools officially sanctioning ratings with real results would\ngive kids the feeling that they really do have a stake in the process, rather\nthan simply being its victims.\n\n4\\. High school is a more structured environment where the process details\nmatter a lot more. So a teacher eg giving out an incomplete homework problem\nis actually a valid indictment rather than the stressed out nitpicking of a\nculture shocked freshman.\n\n5\\. In college, there's a certain level of appreciation for the material that\neveryone should have but doesn't necessarily, causing them to get frustrated\nat a professor with a dry personality. Whereas with high school, the idea is\nthat everybody should be learning a cursory understanding of all subjects.\n\n6\\. In college, there's a huge variation in the level of courses. One specific\nprofessor I had for a seminar where it was basically his PhD research group\nand me, an undergrad who'd just started on a simultaneous master's. I learned\n_a lot_ in that class, and really appreciated him. I then ended up in a grad-\nlevel \"intro\" course with him (which I knew was an utter waste going in, but\nit was the only thing that fit my schedule). Most of the students were rote-\nmemorization paying-for-credential types, but his style certainly did them no\nfavors either, and I can definitely see my recollection echoed in a few of his\ncurrent reviews. I'd say that he's still a teaching asset, but not for intro\nlectures where most students aren't already committed to the subject.\n\nReally, there just needs to be _some_ extrinsic motivation\/reward for teachers\nthat are truly making a difference versus simply clock-punching, and that's\nnot more top-down testing edicts that further shackle them. And sure, the\nimmediate reaction shouldn't be to fire the lowest-reviewed, but neither\nshould we pretend that they deserve similar compensation to the exceptional\nones.\n\n------\ncarsongross\nYou think you know what field workers do. Right? Wrong.\n\nYou think you know what factory workers do. Right? Wrong.\n\nYou think you know what farmers do. Right? Wrong.\n\nYou think you know what oil rig operators do. Right? Wrong.\n\nYou think you know what coffee shop owners do. Right? Wrong.\n\nYou think you know what lawn care specialists do. Right? Wrong.\n\n~~~\nmandalar12\nI agree with your point: the title is sensationalist. The difference between\nteaching and owning a coffee shop (and the others examples) is that few people\nwill try to tell you how to handle your coffee shop while a lot think they\nknow better than you how their children should be taught.\n\n~~~\ncarsongross\nI've seen a close friend work 20 hours a day, barely make payroll, deal with\nemployee drug habits and try to minimize the legal damage a sociopathic\nemployee did.\n\nYou don't know what it's like owning a coffee shop.\n\n------\njerf\nIn other words, teachers are human and have real lives. This may be news to an\n18-year-old, but I'd really be surprised if it's really news to that many\npeople above 30. I may not be a teacher but I could fill a very stylistically-\nsimilar paragraph or two with the woes that have befallen me, too. Most people\ncan.\n\nThis strikes me as a variant on the _You don 't know what's like!_ meme... as\na rule of thumb, you should _never_ say that to anybody. You have no idea what\nthey've been through. Everyone you pass on the street has a story, and no\nmatter how bad you think yours is, you've got no guarantee that they don't\nhave one worse than you.\n\nWhat this essay describes is not specially \"teaching\", it's _life_.\n\n~~~\nSniperfish\nYou as an individual and your profession as a whole are different.\n\nThere is a very pertinent and legitimate point made in the article that\n-teaching- is not a respected industry.\n\nIt's not exactly a new comment!\n\n~~~\nhumanrebar\nCome to think of it, I don't think I hear teaching described as an industry\nvery often.\n\nWhat would be different if teaching were considered an industry? Would it be\nbetter?\n\n------\nVengefulCynic\nTeaching falls into the same category as stage magic, stand-up comedy and\nwriting - it looks easy and effortless when done by an expert because that's\npart of the expertise. Capturing attention, exciting young minds and engaging\nthem is something that, when done effectively, is transparent because that's\nhow it works best. The whole host of knock-on problems that are spawned by\nthis apparent ease are well-documented in TFA.\n\n------\nrjzzleep\ni see a lot of comments saying that we're watching from the sidelines\ncriticizing, and therefore have no clue what's going on.\n\nHow is that even remotely true? We are the victims of the system. We\nexperience firsthand what they do or believe they do.\n\nThis is like saying you think you know what the TSA is doing. Right? Wrong. Of\ncourse we do, we're the ones being screened.\n\nwhat we don't know is the logic and culture behind the decisions we see, but\nthat doesn't take any right away to criticize it.\n\nhaving been an overachiever in school, and early university, it's been a\nconstant struggle. \"oh but school is not actually made for people like you\"\nyou say. yeah, i know. how is that not a problem?\n\nedit: don't get me wrong, i've had a few really good teachers. but they've\nbeen rather few. and no, i'm not just counting the teachers i liked as good.\n\n~~~\nlewispollard\nDoes someone who's used a computer all their life know the ins and outs of\nbeing a programmer? Would you listen to their recommendations on how to\nimprove your code? The answer is likely yes, feedback from customers is\nimportant - but you're not gonna get any useful advice re: the architecture or\nthe design patterns used.\n\n------\ndanso\nAwhile ago, I had a teacher for a roommmate, and one who was young and very\npassionate, and I hope, good at it, because we were best friends and I'd hate\nto think I'd be a poor judge :). But I rarely heard her talk about the pure\njoy of teaching, at least compared to the difficulties of dealing with the\nmanagement (the principal) and other logistics issues...such as having to pay\nfor her own classroom supplies, including books that she wanted if they\nweren't on the state-wide curriculum, and pencil and paper for her poorer\nstudents.\n\nHer complaints about office politics were what really surprised me. Even\nthough I know every bureaucracy is universally crushing (well, maybe I grok\nbetter now after watching The Wire), it just seems that being a great,\npassionate teacher, supersedes any kind of office bullshit...such as the way\nprincipal communicates with you. But then again, if you can't get along with\nthe person who runs the place, and you're put in a shitty classroom and have\nto share a teacher's officespace with 3 other novices...how could that _not_\naffect your teaching performance and job satisfaction?\n\nOne memory I still have from high school was one afternoon when I had to stay\nafter school to give a presentation to the teachers on their regular Thursday-\nschool-wide meeting. The meeting was in the cafeteria...and you know how lunch\ntables reflect a sort of social-hierarchy among kids? It was no different for\nthe teachers...and even more surprising, the social lines seemed to fall along\nwith how I, as a student, expected them to (attractive young teachers sat with\nthe other young teachers; cool popular teachers could sit anywhere they want;\nthe weird chemistry teacher sat in the corner). I mean, it's one thing to have\nperceptions as a kid, but I _knew_ I was a petty kid...so it was a surprise to\nsee that things were not much different in the adult world.\n\n~~~\narbitrage\nGrok means to understand in fullness ... from the Heinlein novel, the\netymology of the word comes from to drink or to consume.\n\nYou cannot grok something just by watching it.\n\n~~~\ndanso\nYeah, but we're talking about _The Wire_ here :). But also I was an education\nreporter, worked as an aide, and have been part of other bureaucracies\nmyself...\n\n------\nnawitus\n>Most of all, we need to stop thinking that we know anything about teaching\nmerely by virtue of having once been students.\n\nI know something about teaching by reading peer-reviewed studies which give\nevidence for better teaching methods, but are almost never adopted because the\nteaching systems and\/or teachers are extremely conservative apparently all\naround the world.\n\nIn fact, I'd trust studies over teachers any day.\n\n------\nlarrik\nI feel like you could write this about basically any profession, besides the\nusual \"teachers are underpaid\" rant.\n\n------\nhumanrebar\n\n > All of you former students: you did not design\n > curricula, plan lessons, attend faculty meetings,\n > assess papers, design rubrics, create exams, prepare \n > report cards, and monitor attendance. You did\n > not tutor students, review rough drafts, and create\n > study questions. You did not assign homework. You\n > did not write daily lesson objectives on the\n > white board. You did not write poems of the week\n > on the white board. You did not write homework on\n > the white board. You did not learn to write\n > legibly on the white board while simultaneously\n > making sure that none of your students threw a\n > chair out a window.\n \n\nI'm not a teacher, so I could be wrong, but it seems to me that much of this\nlist falls into two categories:\n\n1\\. Routine things that could be orders of magnitude more efficient (or even\nfully automated) given enough resources. In most cases, the resources needed\nwould be fairly modest compared to the aggregate amount of effort teachers\neverywhere spend on them. Writing and grading elementary-level math tests, for\nexample, shouldn't take any time at all given the right software.\n\n2\\. Routine things that couldn't be automated well but could easily be done by\nsome sort of entry-level assistant. Babysitting and discipline tasks don't\nrequire college degrees.\n\nIt strikes me that the economics of education are structured in a way that\nthere is marginal impetus to improve efficiencies in the day-to-day work of\nteachers.\n\n~~~\nhackluck\nYou are not a teacher. And from your comments, you have not looked too much in\nthe research about how to teach students.\n\nYes, certain things COULD be automated... at considerable expense to student\nachievement. One big thing they have found - remove the personal feedback and\nconnection to students --> lose the motivation of students. If a teacher (the\nsame teacher) isn't interacting with a student consistently at nearly every\nstep of the learning process, the feedback doesn't stick and the student loses\nmotivation.\n\nIt would be interesting to looking to the basic research behind the feedback-\nachievement connection and stereotype threat to start.\n\nHope that helps you address some of the problems with the automate\/delegate\nsolutions so often thrown at teachers.\n\n~~~\nhumanrebar\n> Yes, certain things COULD be automated... at considerable expense to student\n> achievement.\n\nI seriously doubt that letting teachers automatically grade arithmetic tests\nwill hurt student achievement. The fact is that many teachers do that sort of\nthing at home in what should be considered overtime hours. I would like to\nhear how automatic grading causes student achievement to suffer.\n\nLikewise, I'm skeptical that it should be solely educators' responsibility to\nmake sure chairs are not being thrown out windows. Letting teachers focus on\neducating and not babysitting seems like a good thing.\n\n------\ncarlmcqueen\nI did two years of a special ed major in college before switching over to\ncomputer science and I can say that the ed program I was in covered in depth\nhow to teach and handle a class room, it focused on how to teach math to\npeople who don't understand any concepts, and the department had additional\noffered classes if you wanted to do teach for america or inner city schooling.\n\nSpeaking with friends who have become professors they are often jealous of\nthis because they were never given any kind of 'teaching' classes. Their under\ngrad wasn't in education and their teaching experience was trial by fire\nteaching assistant jobs of handling undergrad college courses.\n\nAll that said, I grow tired of the arguments and articles of 'don't speak\nunless you've walked a thousand miles' which I felt as I read this article.\nNot all knowledge and understanding must derive from doing something to have a\nvalid opinion. We need to treat teachers better and find better pay structures\nbut I've found no harsher critics of teachers and our schools than the\nteachers I went to college with as they filter into the systems and find tired\nand broken systems in which they get no voice until they have 'tenure'.\n\n~~~\nhackluck\n> All that said, I grow tired of the arguments and articles > of 'don't speak\n> unless you've walked a thousand miles' > which I felt as I read this\n> article. Not all knowledge and > understanding must derive from doing\n> something to have a > valid opinion. We need to treat teachers better and\n> find > better pay structures but I've found no harsher critics of > teachers\n> and our schools than the teachers I went to > college with as they filter\n> into the systems and find > tired and broken systems in which they get no\n> voice until > they have 'tenure'.\n\nI don't know that this article is so much about \"not speaking bad about\nteachers\", but about having compassion for teachers and talking about\neducation with a little more humility for the institution that helped produce\nyou. I would say this article is more of the \"teachers don't write articles\nabout how to __________ better, so don't let _______________ers tell teachers\nhow to teach better\" variety.\n\nAnd I totally agree that most young teachers are completely overwhelmed by the\nridiculous systems in which they are forced to teach. My only hope is that\nsome of these young, inspiring teachers remain in the profession long enough\nto change the broken system (which might take a LONG time!). The unions are\nbroken; for the most part, teachers are not.\n\n------\nhueving\nI'm not sure what the implication at the end is about public policy? Even if\nwe supposedly do not understand teaching, that does not mean we can't form\nopinions about the current system and develop policies for it. That's\nprecisely how politics work in every other field.\n\nHow many people who want to ban fracking actually understand fracking or\nprecisely what the real risks are? How many people want to ban nuclear energy\nand don't understand any of the actual risks of modern nuclear power plants?\n\nPolitics suck for anyone that isn't a politician. Each industry must learn how\nto deal with that aspect. Writing an appeal to emotion on the Washington Post\nis not going to sway anyone. It just resonates with people already on their\nside and sounds like whining to people that aren't.\n\n------\nnilkn\nSince the author says she started out making 5 times as much as a lawyer than\nas a teacher, I can only assume she landed one of the associate jobs at a\nmajor law firm straight out of law school making $160k+.\n\nShe makes it sound like anybody can hit up law school and come out making\nalmost $200k. The vast majority of law graduates do not land jobs like that.\nThe vast majority also have nearly crippling debt. The vast majority of the\nfirms paying $160k+ are also in hyper expensive metro areas, whereas teachers\ncan live comfortable lives in very rural towns (if they want to).\n\n~~~\nnumo16\nWhile what she is stating might not be the case for all law school grads, it\nisn't as far fetched to come out of school easily making 2-3 times what a\nstarting teacher is making, depepnding on the degree you choose. As a software\nengineer in michigan, you can come out of uni with a BS in computer science\nand easily find a job paying $55k+ and make 2-3 times as much as a starting\nteacher ($35k if you're lucky and find a good school that has funding) in the\nsame state after a year or two of experience.\n\n------\nfuse117\nThis story strikes home with me. Like the author, I too picked up an MAT,\ntaught for a couple years, and then left the field to pursue other\nopportunities. In the 3-4 years since I left, I have worked a lot less, made a\nlot more, and feel much more respected in what I do.\n\n------\nbiesnecker\n\"You think you know what teachers do. Right? Wrong.\"\n\nSo I'm wrong that I think that I know what teachers do?\n\nDo teachers teach you how to write intelligible headlines?\n\n------\nsmoyer\nMy wife teaches classes at a local high school as well as the university here\nshe's successful because she works hard at it, has a natural aptitude to teach\nand she cares about her subjects and shows it. I think I had 3-4 outstanding\nteachers during my 13 years in public school and they all had these same\ncharacteristics. I had plenty of bad teaches too.\n\n------\nPakG1\nMy cousin is a high school teacher and posted this article on Facebook with\nthe comment that it's like saying just because you had parents, you think you\nknow everything there is to know about parenting.\n\n------\nthe_watcher\nThis is just a painful argument to authority.\n\n------\ntokenadult\nEducation policy is the issue that drew me to participate on Hacker News,[1]\nso I'll jump in here too. I get the impression that mathattack, whose comments\nI enjoy reading, may have posted this article for disagreement. The Answer\nSheet blog from which this guest post comes is basically a propaganda organ,\nand some of the guest posts from the same blog that were submitted to Hacker\nNews in the past were exposed as hack jobs after discussion here.[2]\n\nThe obligatory disclosure here is to note that I am a classroom teacher by\noccupation. Over the years, I have been a teacher of Chinese to native\nspeakers of English, a teacher of English to native speakers of Chinese (and\nother languages), and most recently a teacher of advanced elementary\nmathematics (\"prealgebra\" mathematics for third-, fourth-, and fifth-graders)\nfor a nonprofit organization in my town. My HN user profile describes a bit\nmore of my background.\n\nYep, classroom teaching is hard, no doubt about it. It has emotional rewards\nthat some people value highly enough that it is a sought-after occupation, not\na labor-shortage occupation, and that has the most to do with teacher\ncompensation. Classroom teaching by teachers in private practice (like me) can\nalso be poorly compensated (relative to the difficulty of doing the job well)\nbecause most clients have already paid for \"free\" lessons at the local public\nschools through their taxes, and will only pay out of pocket for a private\nlesson if it is truly superior in some way. \"In modern times [as contrasted\nwith ancient times] the diligence of public teachers is more or less corrupted\nby the circumstances which render them more or less independent of their\nsuccess and reputation in their particular professions. Their salaries, too,\nput the private teacher, who would pretend to come into competition with them,\nin the same state with a merchant who attempts to trade without a bounty in\ncompetition with those who trade with a considerable one. . . . The privileges\nof graduation, besides, are in many countries . . . obtained only by attending\nthe lectures of the public teachers. . . . The endowment of schools and\ncolleges have, in this manner, not only corrupted the diligence of public\nteachers, but have rendered it almost impossible to have any good private\nones.\" \\-- Adam Smith, The Wealth of Nations, Book V, Part 3, Article II\n(1776)\n\nA couple of the comments posted here before I arrived in the thread mention\nthe particular skills that a teacher needs to have to teach a class\neffectively. There is much interesting research on this coming from the\ncharter school movement, with some of the best how-to research coming from the\nTeach Like a Champion[3] project. I love learning about new ways to be a more\neffective teacher. Besides actual teacher skills, another grave problem in\nUnited States school is extremely poor teaching materials[4] and I devote\nhundreds of hours to curriculum planning and seeking out the best available\ntextbooks[5] for the subjects I teach.\n\nA good teacher is worth a lot.[6] We would not go far wrong by saying that a\ngood teacher is literally worth his or her weight in gold. But the tricky\nissue in school administration is distinguishing effective from ineffective\nteachers. To ensure that school leaders have incentives to find and reward the\nbest teachers, we need to make sure that learners (or the adult guardians of\nminor learners) have the power to shop, the power to refuse the services of an\nineffective teacher and to seek out the services of an effective teacher.\nTeachers will gain both more pay and more respect if learners gain power to\nshop.\n\n[1]\n[http:\/\/news.ycombinator.com\/item?id=4728123](http:\/\/news.ycombinator.com\/item?id=4728123)\n\n[2]\n[https:\/\/news.ycombinator.com\/item?id=3327847](https:\/\/news.ycombinator.com\/item?id=3327847)\n\n[3] [http:\/\/teachlikeachampion.com\/](http:\/\/teachlikeachampion.com\/)\n\n[4]\n[http:\/\/open.salon.com\/blog\/annie_keeghan\/2012\/02\/17\/afraid_o...](http:\/\/open.salon.com\/blog\/annie_keeghan\/2012\/02\/17\/afraid_of_your_childs_math_textbook_you_should_be)\n\n[5]\n[http:\/\/www.artofproblemsolving.com\/Store\/viewitem.php?item=p...](http:\/\/www.artofproblemsolving.com\/Store\/viewitem.php?item=prealgebra)\n\n[6] [http:\/\/hanushek.stanford.edu\/publications\/valuing-\nteachers-h...](http:\/\/hanushek.stanford.edu\/publications\/valuing-teachers-how-\nmuch-good-teacher-worth)\n\n","meta":"{'id': '7318922'}"}
diff --git a/notebooks/tutorials/data-engineer/01-setting-up-dev-mode.ipynb b/notebooks/tutorials/data-engineer/01-setting-up-dev-mode.ipynb
index ac6c51d776b..3e676575486 100644
--- a/notebooks/tutorials/data-engineer/01-setting-up-dev-mode.ipynb
+++ b/notebooks/tutorials/data-engineer/01-setting-up-dev-mode.ipynb
@@ -321,6 +321,14 @@
"source": [
"node.land()"
]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4624a381",
+ "metadata": {},
+ "outputs": [],
+ "source": []
}
],
"metadata": {
@@ -339,7 +347,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.2"
+ "version": "3.9.16"
},
"toc": {
"base_numbering": 1,
diff --git a/notebooks/tutorials/enclaves/Enclave-single-notebook-DO-DS.ipynb b/notebooks/tutorials/enclaves/Enclave-single-notebook-DO-DS.ipynb
index 3b6a34e3cb7..adfc247e21a 100644
--- a/notebooks/tutorials/enclaves/Enclave-single-notebook-DO-DS.ipynb
+++ b/notebooks/tutorials/enclaves/Enclave-single-notebook-DO-DS.ipynb
@@ -427,7 +427,8 @@
"metadata": {},
"outputs": [],
"source": [
- "@sy.syft_function_single_use(canada_census_data=canada_census_data, italy_census_data=italy_census_data, share_results_with_owners=True)\n",
+ "@sy.syft_function_single_use(canada_census_data=canada_census_data, italy_census_data=italy_census_data,\n",
+ " share_results_with_owners=True)\n",
"def compute_census_matches(canada_census_data, italy_census_data):\n",
" import recordlinkage\n",
" \n",
@@ -687,7 +688,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.10.8"
+ "version": "3.9.16"
},
"toc": {
"base_numbering": 1,
diff --git a/notebooks/tutorials/enclaves/Enclave-single-notebook-high-low-network.ipynb b/notebooks/tutorials/enclaves/Enclave-single-notebook-high-low-network.ipynb
index b14f701cee6..13405bc9807 100644
--- a/notebooks/tutorials/enclaves/Enclave-single-notebook-high-low-network.ipynb
+++ b/notebooks/tutorials/enclaves/Enclave-single-notebook-high-low-network.ipynb
@@ -136,21 +136,6 @@
")"
]
},
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "fe8e5855",
- "metadata": {},
- "outputs": [],
- "source": [
- "ca_node_high = sy.Orchestra.launch(\n",
- " name=\"canada-2\",\n",
- " local_db=True,\n",
- " reset=True,\n",
- "# enable_warnings=True,\n",
- ")"
- ]
- },
{
"cell_type": "code",
"execution_count": null,
@@ -1001,14 +986,6 @@
"real_result"
]
},
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ec88a993",
- "metadata": {},
- "outputs": [],
- "source": []
- },
{
"cell_type": "code",
"execution_count": null,
@@ -1034,7 +1011,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.10.8"
+ "version": "3.9.16"
},
"toc": {
"base_numbering": 1,
diff --git a/notebooks/tutorials/hello-syft/01-hello-syft.ipynb b/notebooks/tutorials/hello-syft/01-hello-syft.ipynb
index 01f89edc93f..f9af36a4fce 100644
--- a/notebooks/tutorials/hello-syft/01-hello-syft.ipynb
+++ b/notebooks/tutorials/hello-syft/01-hello-syft.ipynb
@@ -548,7 +548,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.10.8"
+ "version": "3.9.16"
},
"toc": {
"base_numbering": 1,
diff --git a/notebooks/tutorials/pandas-cookbook/01-reading-from-a-csv.ipynb b/notebooks/tutorials/pandas-cookbook/01-reading-from-a-csv.ipynb
index 4a8e63dd094..07d7d866e67 100644
--- a/notebooks/tutorials/pandas-cookbook/01-reading-from-a-csv.ipynb
+++ b/notebooks/tutorials/pandas-cookbook/01-reading-from-a-csv.ipynb
@@ -839,7 +839,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.5"
+ "version": "3.9.16"
},
"toc": {
"base_numbering": 1,
diff --git a/notebooks/tutorials/pandas-cookbook/02-selecting-data-finding-common-complain.ipynb b/notebooks/tutorials/pandas-cookbook/02-selecting-data-finding-common-complain.ipynb
index ecb945878a1..ed4599f8f62 100644
--- a/notebooks/tutorials/pandas-cookbook/02-selecting-data-finding-common-complain.ipynb
+++ b/notebooks/tutorials/pandas-cookbook/02-selecting-data-finding-common-complain.ipynb
@@ -992,7 +992,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.10.8"
+ "version": "3.9.16"
},
"toc": {
"base_numbering": 1,
diff --git a/packages/grid/backend/grid/core/config.py b/packages/grid/backend/grid/core/config.py
index 7f974061e6a..e5b34d75a42 100644
--- a/packages/grid/backend/grid/core/config.py
+++ b/packages/grid/backend/grid/core/config.py
@@ -118,6 +118,7 @@ def get_emails_enabled(cls, v: bool, values: Dict[str, Any]) -> bool:
S3_PRESIGNED_TIMEOUT_SECS: int = int(
os.getenv("S3_PRESIGNED_TIMEOUT_SECS", 1800)
) # 30 minutes in seconds
+ SEAWEED_MOUNT_PORT: int = int(os.getenv("SEAWEED_MOUNT_PORT", 4001))
REDIS_HOST: str = str(os.getenv("REDIS_HOST", "redis"))
REDIS_PORT: int = int(os.getenv("REDIS_PORT", 6379))
@@ -132,6 +133,13 @@ def get_emails_enabled(cls, v: bool, values: Dict[str, Any]) -> bool:
MONGO_PORT: int = int(os.getenv("MONGO_PORT", 0))
MONGO_USERNAME: str = str(os.getenv("MONGO_USERNAME", ""))
MONGO_PASSWORD: str = str(os.getenv("MONGO_PASSWORD", ""))
+ DEV_MODE: bool = True if os.getenv("DEV_MODE", "false").lower() == "true" else False
+ # ZMQ stuff
+ QUEUE_PORT: int = int(os.getenv("QUEUE_PORT", 0))
+ CREATE_PRODUCER: bool = (
+ True if os.getenv("CREATE_PRODUCER", "false").lower() == "true" else False
+ )
+ N_CONSUMERS: int = int(os.getenv("N_CONSUMERS", 0))
SQLITE_PATH: str = os.path.expandvars("$HOME/data/db/")
SINGLE_CONTAINER_MODE: bool = str_to_bool(os.getenv("SINGLE_CONTAINER_MODE", False))
@@ -139,7 +147,6 @@ def get_emails_enabled(cls, v: bool, values: Dict[str, Any]) -> bool:
True if os.getenv("TEST_MODE", "false").lower() == "true" else False
)
ASSOCIATION_TIMEOUT: int = 10
- DEV_MODE: bool = True if os.getenv("DEV_MODE", "false").lower() == "true" else False
class Config:
case_sensitive = True
diff --git a/packages/grid/backend/grid/core/node.py b/packages/grid/backend/grid/core/node.py
index 19d22c80567..0a1a32fa814 100644
--- a/packages/grid/backend/grid/core/node.py
+++ b/packages/grid/backend/grid/core/node.py
@@ -8,6 +8,8 @@
from syft.node.node import get_node_side_type
from syft.node.node import get_node_type
from syft.node.node import get_node_uid_env
+from syft.service.queue.zmq_queue import ZMQClientConfig
+from syft.service.queue.zmq_queue import ZMQQueueConfig
from syft.store.blob_storage.seaweedfs import SeaweedFSClientConfig
from syft.store.blob_storage.seaweedfs import SeaweedFSConfig
from syft.store.mongo_client import MongoStoreClientConfig
@@ -19,6 +21,17 @@
from grid.core.config import settings
+def queue_config() -> ZMQQueueConfig:
+ queue_config = ZMQQueueConfig(
+ client_config=ZMQClientConfig(
+ create_producer=settings.CREATE_PRODUCER,
+ queue_port=settings.QUEUE_PORT,
+ n_consumers=settings.N_CONSUMERS,
+ )
+ )
+ return queue_config
+
+
def mongo_store_config() -> MongoStoreConfig:
mongo_client_config = MongoStoreClientConfig(
hostname=settings.MONGO_HOST,
@@ -42,7 +55,8 @@ def seaweedfs_config() -> SeaweedFSConfig:
access_key=settings.S3_ROOT_USER,
secret_key=settings.S3_ROOT_PWD,
region=settings.S3_REGION,
- bucket_name=get_node_uid_env(),
+ default_bucket_name=get_node_uid_env(),
+ mount_port=settings.SEAWEED_MOUNT_PORT,
)
return SeaweedFSConfig(client_config=seaweed_client_config)
@@ -63,9 +77,9 @@ def seaweedfs_config() -> SeaweedFSConfig:
worker_class = worker_classes[node_type]
single_container_mode = settings.SINGLE_CONTAINER_MODE
-
store_config = sql_store_config() if single_container_mode else mongo_store_config()
blob_storage_config = None if single_container_mode else seaweedfs_config()
+queue_config = queue_config()
worker = worker_class(
name=node_name,
@@ -75,4 +89,6 @@ def seaweedfs_config() -> SeaweedFSConfig:
enable_warnings=enable_warnings,
blob_storage_config=blob_storage_config,
local_db=single_container_mode,
+ queue_config=queue_config,
+ migrate=True,
)
diff --git a/packages/grid/backend/grid/main.py b/packages/grid/backend/grid/main.py
index 8c4b34507d8..e24bf4a95cf 100644
--- a/packages/grid/backend/grid/main.py
+++ b/packages/grid/backend/grid/main.py
@@ -32,6 +32,7 @@
)
app.include_router(api_router, prefix=settings.API_V2_STR)
+print("Included routes, app should now be reachable")
if settings.DEV_MODE:
diff --git a/packages/grid/backend/grid/start.sh b/packages/grid/backend/grid/start.sh
index 97de442333b..83ce1d31f7c 100755
--- a/packages/grid/backend/grid/start.sh
+++ b/packages/grid/backend/grid/start.sh
@@ -1,9 +1,10 @@
#! /usr/bin/env bash
set -e
+pip install nltk
echo "Running start.sh with RELEASE=${RELEASE} and $(id)"
-
export GEVENT_MONKEYPATCH="False"
+
APP_MODULE=grid.main:app
LOG_LEVEL=${LOG_LEVEL:-info}
HOST=${HOST:-0.0.0.0}
diff --git a/packages/grid/default.env b/packages/grid/default.env
index 580df9a2722..4719e3da615 100644
--- a/packages/grid/default.env
+++ b/packages/grid/default.env
@@ -34,6 +34,8 @@ STACK_API_KEY=""
# Backend
BACKEND_CORS_ORIGINS='["http://localhost","http://localhost:4200","http://localhost:3000","http://localhost:8080","https://localhost","https://localhost:4200","https://localhost:3000","https://localhost:8080","http://dev.grid.openmined.org","https://stag.grid.openmined.org","https://grid.openmined.org"]'
+BACKEND_STORAGE_PATH=credentials-data
+SEAWEED_MOUNT_PORT=4001
PROJECT_NAME=grid
SECRET_KEY=changethis
DEFAULT_ROOT_EMAIL=info@openmined.org
@@ -50,6 +52,9 @@ DOMAIN_CHECK_INTERVAL=60
ASSOCIATION_TIMEOUT=10
USERS_OPEN_REGISTRATION=False
DEV_MODE=False
+QUEUE_PORT=5556
+CREATE_PRODUCER=False
+N_CONSUMERS=0
# New Service Flag
USE_NEW_SERVICE=False
diff --git a/packages/grid/docker-compose.build.yml b/packages/grid/docker-compose.build.yml
index a6408749c3b..cc967abc40b 100644
--- a/packages/grid/docker-compose.build.yml
+++ b/packages/grid/docker-compose.build.yml
@@ -15,7 +15,10 @@ services:
target: "backend"
profiles:
- backend
-
+ seaweedfs:
+ build:
+ context: ${RELATIVE_PATH}./seaweedfs
+ dockerfile: seaweedfs.dockerfile
worker:
build:
context: ${RELATIVE_PATH}../
diff --git a/packages/grid/docker-compose.dev.yml b/packages/grid/docker-compose.dev.yml
index badc3fdfafb..b5b41eedfd2 100644
--- a/packages/grid/docker-compose.dev.yml
+++ b/packages/grid/docker-compose.dev.yml
@@ -60,6 +60,7 @@ services:
- ${RELATIVE_PATH}./data/package-cache:/root/.cache
environment:
- DEV_MODE=True
+ - WATCHFILES_FORCE_POLLING=true
stdin_open: true
tty: true
@@ -82,8 +83,8 @@ services:
seaweedfs:
profiles:
- blob-storage
- # volumes:
- # - ./data/seaweedfs:/data
+ volumes:
+ - ./data/seaweedfs:/data
ports:
- "9333" # admin web port
- "8888" # filer web port
diff --git a/packages/grid/docker-compose.yml b/packages/grid/docker-compose.yml
index b6da43dc9b4..12f5ff56300 100644
--- a/packages/grid/docker-compose.yml
+++ b/packages/grid/docker-compose.yml
@@ -122,6 +122,7 @@ services:
- backend
depends_on:
- proxy
+ - mongo
env_file:
- .env
environment:
@@ -147,9 +148,17 @@ services:
- ENABLE_OBLV=${ENABLE_OBLV}
- DEFAULT_ROOT_EMAIL=${DEFAULT_ROOT_EMAIL}
- DEFAULT_ROOT_PASSWORD=${DEFAULT_ROOT_PASSWORD}
+ - BACKEND_STORAGE_PATH=${BACKEND_STORAGE_PATH}
+ - QUEUE_PORT=${QUEUE_PORT}
+ - CREATE_PRODUCER=true
+ - N_CONSUMERS=0
+ - HOST_GRID_PATH=${PWD}
+ command: "./grid/start.sh"
network_mode: service:proxy
volumes:
+ - ${BACKEND_STORAGE_PATH}:/storage
- ${CREDENTIALS_VOLUME}:/root/data/creds/
+ - /var/run/docker.sock:/var/run/docker.sock
stdin_open: true
tty: true
labels:
@@ -227,18 +236,19 @@ services:
- blob-storage
depends_on:
- proxy
- image: "${DOCKER_IMAGE_SEAWEEDFS?Variable not set}:${SEAWEEDFS_VERSION}"
+ image: "${DOCKER_IMAGE_SEAWEEDFS?Variable not set}:${VERSION-latest}"
environment:
- S3_VOLUME_SIZE_MB=${S3_VOLUME_SIZE_MB:-1024}
- S3_ROOT_USER=${S3_ROOT_USER:-admin}
- S3_ROOT_PWD=${S3_ROOT_PWD:-admin}
- S3_PORT=${S3_PORT:-8888}
+ - SEAWEED_MOUNT_PORT=${SEAWEED_MOUNT_PORT:-4001}
+ - STACK_API_KEY=$STACK_API_KEY
entrypoint: ["/bin/sh"]
command:
- - "/etc/seaweedfs/start.sh"
+ - "/start.sh"
volumes:
- - seaweedfs-data:/data/blob
- - seaweedfs-data-2:/data
+ - seaweedfs-data:/data
- ./seaweedfs/filer.toml:/etc/seaweedfs/filer.toml
- ./seaweedfs/start.sh:/etc/seaweedfs/start.sh
labels:
@@ -289,9 +299,6 @@ volumes:
seaweedfs-data:
labels:
orgs.openmined.syft: "this is a syft seaweedfs volume"
- seaweedfs-data-2:
- labels:
- orgs.openmined.syft: "this is a syft seaweedfs volume"
mongo-data:
labels:
orgs.openmined.syft: "this is a syft mongo volume"
diff --git a/packages/grid/seaweedfs/app.py b/packages/grid/seaweedfs/app.py
new file mode 100644
index 00000000000..130c32c360d
--- /dev/null
+++ b/packages/grid/seaweedfs/app.py
@@ -0,0 +1,34 @@
+# type: ignore
+# stdlib
+import json
+import subprocess
+
+# third party
+from flask import Flask
+from flask import request
+
+# Flask application instance
+app = Flask(__name__)
+
+
+@app.route("/configure_azure", methods=["POST"])
+def configure_azure() -> str:
+ first_res = json.loads(request.data.decode("utf-8").replace("'", '"'))
+ account_name = first_res["account_name"]
+ account_key = first_res["account_key"]
+ container_name = first_res["container_name"]
+ remote_name = first_res["remote_name"]
+ bucket_name = first_res["bucket_name"]
+
+ res = subprocess.run(
+ [
+ "bash",
+ "mount_command.sh",
+ remote_name,
+ account_name,
+ bucket_name,
+ container_name,
+ account_key,
+ ]
+ )
+ return str(res.returncode)
diff --git a/packages/grid/seaweedfs/mount_command.sh b/packages/grid/seaweedfs/mount_command.sh
new file mode 100644
index 00000000000..9b4683cc3e5
--- /dev/null
+++ b/packages/grid/seaweedfs/mount_command.sh
@@ -0,0 +1,9 @@
+echo "remote.configure -name=$1 -type=azure -azure.account_name=$2 \
+ -azure.account_key=$5" \
+ | weed shell
+
+echo "s3.bucket.create -name $3" | weed shell
+
+echo "remote.mount -dir=/buckets/$3 -remote=$1/$4" | weed shell
+
+weed filer.remote.sync
diff --git a/packages/grid/seaweedfs/requirements.txt b/packages/grid/seaweedfs/requirements.txt
new file mode 100644
index 00000000000..a94b42bbd06
--- /dev/null
+++ b/packages/grid/seaweedfs/requirements.txt
@@ -0,0 +1,2 @@
+flask==2.3.2
+flask_shell2http==1.9.1
diff --git a/packages/grid/seaweedfs/seaweedfs.dockerfile b/packages/grid/seaweedfs/seaweedfs.dockerfile
new file mode 100644
index 00000000000..224041522ca
--- /dev/null
+++ b/packages/grid/seaweedfs/seaweedfs.dockerfile
@@ -0,0 +1,18 @@
+FROM chrislusf/seaweedfs:3.57
+
+WORKDIR /
+
+RUN apk update && apk upgrade --available
+RUN apk add --no-cache python3 py3-pip ca-certificates bash
+
+COPY ./requirements.txt /requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY ./start.sh /start.sh
+COPY ./mount_command.sh /mount_command.sh
+COPY ./app.py /app.py
+
+RUN chmod +x /start.sh
+RUN chmod +x /mount_command.sh
+
+ENTRYPOINT ["bash", "./start.sh"]
diff --git a/packages/grid/seaweedfs/start.sh b/packages/grid/seaweedfs/start.sh
index d6dc34f535d..f084ec521d7 100644
--- a/packages/grid/seaweedfs/start.sh
+++ b/packages/grid/seaweedfs/start.sh
@@ -1,6 +1,10 @@
-#! /usr/bin/env bash
+#!/usr/bin/env bash
-sleep 30 &&
-echo "s3.configure -access_key ${S3_ROOT_USER} -secret_key ${S3_ROOT_PWD} -user iam -actions Read,Write,List,Tagging,Admin -apply" \
-| weed shell > /dev/null 2>&1 \
-& weed server -s3 -s3.port=${S3_PORT} -master.volumeSizeLimitMB=${S3_VOLUME_SIZE_MB}
\ No newline at end of file
+echo "got api key"
+echo ${STACK_API_KEY}
+export STACK_API_KEY=${STACK_API_KEY}
+
+echo "s3.configure -access_key $S3_ROOT_USER -secret_key $S3_ROOT_PWD \
+-user iam -actions Read,Write,List,Tagging,Admin -apply" | weed shell > /dev/null 2>&1 &
+weed server -s3 -s3.port=$S3_PORT -volume.max=500 -master.volumeSizeLimitMB=$S3_VOLUME_SIZE_MB &
+flask run -p $SEAWEED_MOUNT_PORT --host=0.0.0.0
diff --git a/packages/grid/traefik/docker/dynamic.yml b/packages/grid/traefik/docker/dynamic.yml
index 1271d6f3f67..9b8923e386f 100644
--- a/packages/grid/traefik/docker/dynamic.yml
+++ b/packages/grid/traefik/docker/dynamic.yml
@@ -16,6 +16,14 @@ http:
loadBalancer:
servers:
- url: "http://seaweedfs:8333"
+ seaweedfsmount:
+ loadBalancer:
+ servers:
+ - url: "http://seaweedfs:4001"
+ headscale:
+ loadBalancer:
+ servers:
+ - url: "http://headscale:8080"
routers:
frontend:
rule: "PathPrefix(`/`)"
@@ -40,6 +48,22 @@ http:
middlewares:
- "blob-storage-url"
- "blob-storage-host"
+ blob-storage-mount:
+ rule: "PathPrefix(`/mount`)"
+ entryPoints:
+ - web
+ - vpn
+ service: "seaweedfsmount"
+ middlewares:
+ - "blob-storage-mount-url"
+ vpn:
+ rule: "PathPrefix(`/vpn`)"
+ entryPoints:
+ - web
+ - vpn
+ service: "headscale"
+ middlewares:
+ - "vpn-url"
ping:
rule: "PathPrefix(`/ping`)"
entryPoints:
@@ -57,3 +81,11 @@ http:
stripprefix:
prefixes: /blob
forceslash: true
+ blob-storage-mount-url:
+ stripprefix:
+ prefixes: /mount
+ forceslash: true
+ vpn-url:
+ stripprefix:
+ prefixes: /vpn
+ forceslash: true
diff --git a/packages/hagrid/hagrid/cli.py b/packages/hagrid/hagrid/cli.py
index 8d26c3b429f..0c14dbeabb9 100644
--- a/packages/hagrid/hagrid/cli.py
+++ b/packages/hagrid/hagrid/cli.py
@@ -1276,7 +1276,7 @@ def create_launch_cmd(
parsed_kwargs["trace"] = False
if ("trace" not in kwargs or kwargs["trace"] is None) and parsed_kwargs["dev"]:
# default to trace on in dev mode
- parsed_kwargs["trace"] = True
+ parsed_kwargs["trace"] = False
elif "trace" in kwargs:
parsed_kwargs["trace"] = str_to_bool(cast(str, kwargs["trace"]))
diff --git a/packages/hagrid/hagrid/orchestra.py b/packages/hagrid/hagrid/orchestra.py
index 273ff441d3d..d8fbd2d6e7e 100644
--- a/packages/hagrid/hagrid/orchestra.py
+++ b/packages/hagrid/hagrid/orchestra.py
@@ -8,14 +8,12 @@
import inspect
import os
import subprocess # nosec
+from threading import Thread
from typing import Any
from typing import Callable
from typing import Optional
from typing import Union
-# third party
-import gevent
-
# relative
from .cli import str_to_bool
from .grammar import find_available_port
@@ -46,7 +44,6 @@ def read_stream(stream: subprocess.PIPE) -> None:
if not line:
break
print(line, end="")
- gevent.sleep(0)
def to_snake_case(name: str) -> str:
@@ -237,6 +234,9 @@ def deploy_to_python(
local_db: bool,
node_side_type: NodeSideType,
enable_warnings: bool,
+ n_consumers: int,
+ create_producer: bool = False,
+ queue_port: Optional[int] = None,
) -> Optional[NodeHandle]:
sy = get_syft_client()
if sy is None:
@@ -264,6 +264,7 @@ def deploy_to_python(
host=host,
port=port,
reset=reset,
+ processes=processes,
dev_mode=dev_mode,
tail=tail,
node_type=node_type_enum,
@@ -296,6 +297,7 @@ def deploy_to_python(
sig = inspect.signature(worker_class.named)
if "node_type" in sig.parameters.keys():
worker = worker_class.named(
+ dev_mode=dev_mode,
name=name,
processes=processes,
reset=reset,
@@ -303,6 +305,10 @@ def deploy_to_python(
node_type=node_type_enum,
node_side_type=node_side_type,
enable_warnings=enable_warnings,
+ n_consumers=n_consumers,
+ create_producer=create_producer,
+ queue_port=queue_port,
+ migrate=True,
)
else:
# syft <= 0.8.1
@@ -314,12 +320,17 @@ def deploy_to_python(
)
else:
raise NotImplementedError(f"node_type: {node_type_enum} is not supported")
+
+ def stop() -> None:
+ worker.stop()
+
return NodeHandle(
node_type=node_type_enum,
deployment_type=deployment_type_enum,
name=name,
python_node=worker,
node_side_type=node_side_type,
+ shutdown=stop,
)
@@ -434,12 +445,14 @@ def deploy_to_container(
process = subprocess.Popen( # nosec
commands, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=env
)
- # Start gevent threads to read and print the output and error streams
- stdout_thread = gevent.spawn(read_stream, process.stdout)
- stderr_thread = gevent.spawn(read_stream, process.stderr)
-
- # Wait for the threads to finish
- gevent.joinall([stdout_thread, stderr_thread], raise_error=True)
+ # Start threads to read and print the output and error streams
+ stdout_thread = Thread(target=read_stream, args=(process.stdout,))
+ stderr_thread = Thread(target=read_stream, args=(process.stderr,))
+ # todo, raise errors
+ stdout_thread.start()
+ stderr_thread.start()
+ stdout_thread.join()
+ stderr_thread.join()
if not cmd:
return NodeHandle(
@@ -474,6 +487,9 @@ def launch(
verbose: bool = False,
render: bool = False,
enable_warnings: bool = False,
+ n_consumers: int = 0,
+ create_producer: bool = False,
+ queue_port: Optional[int] = None,
) -> Optional[NodeHandle]:
if dev_mode is True:
os.environ["DEV_MODE"] = "True"
@@ -516,6 +532,9 @@ def launch(
local_db=local_db,
node_side_type=node_side_type_enum,
enable_warnings=enable_warnings,
+ n_consumers=n_consumers,
+ create_producer=create_producer,
+ queue_port=queue_port,
)
elif deployment_type_enum == DeploymentType.K8S:
@@ -587,7 +606,7 @@ def shutdown(
def reset(name: str, deployment_type_enum: DeploymentType) -> None:
if deployment_type_enum == DeploymentType.PYTHON:
sy = get_syft_client()
- _ = sy.Worker.named(name, processes=1, reset=True) # type: ignore
+ _ = sy.Worker.named(name=name, processes=1, reset=True) # type: ignore
elif (
deployment_type_enum == DeploymentType.CONTAINER_STACK
or deployment_type_enum == DeploymentType.SINGLE_CONTAINER
diff --git a/packages/syft/setup.cfg b/packages/syft/setup.cfg
index 3a769afed13..7a981afb333 100644
--- a/packages/syft/setup.cfg
+++ b/packages/syft/setup.cfg
@@ -50,6 +50,7 @@ syft =
sherlock[redis,filelock]==0.4.1
uvicorn[standard]==0.23.2
fastapi==0.103.2
+ psutil==5.9.6
hagrid>=0.3
itables==1.6.2
safetensors==0.4.0
diff --git a/packages/syft/src/syft/client/client.py b/packages/syft/src/syft/client/client.py
index 8fad938c6ab..d2db4cca983 100644
--- a/packages/syft/src/syft/client/client.py
+++ b/packages/syft/src/syft/client/client.py
@@ -153,7 +153,7 @@ def get_cache_key(self) -> str:
def api_url(self) -> GridURL:
return self.url.with_path(self.routes.ROUTE_API_CALL.value)
- def to_blob_route(self, path: str) -> GridURL:
+ def to_blob_route(self, path: str, **kwargs) -> GridURL:
_path = self.routes.ROUTE_BLOB_STORE.value + path
return self.url.with_path(_path)
@@ -345,6 +345,13 @@ def get_node_metadata(self, credentials: SyftSigningKey) -> NodeMetadataJSON:
else:
return self.node.metadata.to(NodeMetadataJSON)
+ def to_blob_route(self, path: str, host=None) -> GridURL:
+ # TODO: FIX!
+ if host is not None:
+ return GridURL(host_or_ip=host, port=8333).with_path(path)
+ else:
+ return GridURL(port=8333).with_path(path)
+
def get_api(
self, credentials: SyftSigningKey, communication_protocol: int
) -> SyftAPI:
@@ -602,6 +609,12 @@ def exchange_route(self, client: Self) -> Union[SyftSuccess, SyftError]:
return result
+ @property
+ def jobs(self) -> Optional[APIModule]:
+ if self.api.has_service("job"):
+ return self.api.services.job
+ return None
+
@property
def users(self) -> Optional[APIModule]:
if self.api.has_service("user"):
diff --git a/packages/syft/src/syft/client/deploy.py b/packages/syft/src/syft/client/deploy.py
index afce2bc1dc9..bd19895ced5 100644
--- a/packages/syft/src/syft/client/deploy.py
+++ b/packages/syft/src/syft/client/deploy.py
@@ -24,7 +24,8 @@ def import_orchestra() -> Any:
return Orchestra
- except Exception: # nosec
+ except Exception as e: # nosec
+ print(e)
pass
return InstallOrchestra()
diff --git a/packages/syft/src/syft/client/domain_client.py b/packages/syft/src/syft/client/domain_client.py
index 55f46c8de4a..05cbb2294d6 100644
--- a/packages/syft/src/syft/client/domain_client.py
+++ b/packages/syft/src/syft/client/domain_client.py
@@ -159,6 +159,12 @@ def code(self) -> Optional[APIModule]:
return self.api.services.code
return None
+ @property
+ def worker(self) -> Optional[APIModule]:
+ if self.api.has_service("worker"):
+ return self.api.services.worker
+ return None
+
@property
def requests(self) -> Optional[APIModule]:
if self.api.has_service("request"):
diff --git a/packages/syft/src/syft/external/oblv/deployment_client.py b/packages/syft/src/syft/external/oblv/deployment_client.py
index 8a170a5d0ac..6b4c1f6d304 100644
--- a/packages/syft/src/syft/external/oblv/deployment_client.py
+++ b/packages/syft/src/syft/external/oblv/deployment_client.py
@@ -25,7 +25,7 @@
from ...client.client import SyftClient
from ...client.client import login
from ...client.client import login_as_guest
-from ...enclave.metadata import EnclaveMetadata
+from ...client.enclave_client import EnclaveMetadata
from ...serde.serializable import serializable
from ...types.uid import UID
from ...util.util import bcolors
diff --git a/packages/syft/src/syft/gevent_patch.py b/packages/syft/src/syft/gevent_patch.py
index 71805aaf599..2265a2ae713 100644
--- a/packages/syft/src/syft/gevent_patch.py
+++ b/packages/syft/src/syft/gevent_patch.py
@@ -2,9 +2,6 @@
import os
from typing import Optional
-# third party
-from gevent import monkey
-
def str_to_bool(bool_str: Optional[str]) -> bool:
result = False
@@ -36,7 +33,3 @@ def is_notebook() -> bool:
jupyter_notebook = is_notebook()
-
-if jupyter_notebook:
- # print("Patching Gevent in Jupyter")
- monkey.patch_all(thread=False)
diff --git a/packages/syft/src/syft/node/node.py b/packages/syft/src/syft/node/node.py
index 0e7f5e82aa6..49d04b33d61 100644
--- a/packages/syft/src/syft/node/node.py
+++ b/packages/syft/src/syft/node/node.py
@@ -10,6 +10,7 @@
import hashlib
from multiprocessing import current_process
import os
+from pathlib import Path
import subprocess # nosec
import traceback
from typing import Any
@@ -23,9 +24,6 @@
import uuid
# third party
-import gevent
-import gipc
-from gipc.gipc import _GIPCDuplexHandle
from nacl.signing import SigningKey
from result import Err
from result import Result
@@ -64,6 +62,9 @@
from ..service.data_subject.data_subject_service import DataSubjectService
from ..service.dataset.dataset_service import DatasetService
from ..service.enclave.enclave_service import EnclaveService
+from ..service.job.job_service import JobService
+from ..service.job.job_stash import Job
+from ..service.log.log_service import LogService
from ..service.metadata.metadata_service import MetadataService
from ..service.metadata.node_metadata import NodeMetadataV3
from ..service.network.network_service import NetworkService
@@ -71,11 +72,16 @@
from ..service.object_search.migration_state_service import MigrateStateService
from ..service.policy.policy_service import PolicyService
from ..service.project.project_service import ProjectService
+from ..service.queue.base_queue import QueueConsumer
+from ..service.queue.base_queue import QueueProducer
from ..service.queue.queue import APICallMessageHandler
from ..service.queue.queue import QueueManager
+from ..service.queue.queue_service import QueueService
+from ..service.queue.queue_stash import ActionQueueItem
from ..service.queue.queue_stash import QueueItem
from ..service.queue.queue_stash import QueueStash
from ..service.queue.zmq_queue import QueueConfig
+from ..service.queue.zmq_queue import ZMQClientConfig
from ..service.queue.zmq_queue import ZMQQueueConfig
from ..service.request.request_service import RequestService
from ..service.response import SyftError
@@ -90,6 +96,7 @@
from ..service.user.user_roles import ServiceRole
from ..service.user.user_service import UserService
from ..service.user.user_stash import UserStash
+from ..service.worker.worker_service import WorkerService
from ..store.blob_storage import BlobStorageConfig
from ..store.blob_storage.on_disk import OnDiskBlobStorageClientConfig
from ..store.blob_storage.on_disk import OnDiskBlobStorageConfig
@@ -132,6 +139,7 @@ def gipc_decoder(obj_bytes):
NODE_SIDE_TYPE = "NODE_SIDE_TYPE"
DEFAULT_ROOT_EMAIL = "DEFAULT_ROOT_EMAIL"
+DEFAULT_ROOT_USERNAME = "DEFAULT_ROOT_USERNAME"
DEFAULT_ROOT_PASSWORD = "DEFAULT_ROOT_PASSWORD" # nosec
@@ -159,6 +167,10 @@ def get_default_root_email() -> Optional[str]:
return get_env(DEFAULT_ROOT_EMAIL, "info@openmined.org")
+def get_default_root_username() -> Optional[str]:
+ return get_env(DEFAULT_ROOT_USERNAME, "Jane Doe")
+
+
def get_default_root_password() -> Optional[str]:
return get_env(DEFAULT_ROOT_PASSWORD, "changethis") # nosec
@@ -184,6 +196,7 @@ def get_venv_packages() -> str:
node_uid_env = get_node_uid_env()
default_root_email = get_default_root_email()
+default_root_username = get_default_root_username()
default_root_password = get_default_root_password()
@@ -237,6 +250,7 @@ def __init__(
action_store_config: Optional[StoreConfig] = None,
document_store_config: Optional[StoreConfig] = None,
root_email: str = default_root_email,
+ root_username: str = default_root_username,
root_password: str = default_root_password,
processes: int = 0,
is_subprocess: bool = False,
@@ -247,16 +261,19 @@ def __init__(
queue_config: Optional[QueueConfig] = None,
node_side_type: Union[str, NodeSideType] = NodeSideType.HIGH_SIDE,
enable_warnings: bool = False,
+ dev_mode: bool = False,
+ migrate: bool = False,
):
# 🟡 TODO 22: change our ENV variable format and default init args to make this
# less horrible or add some convenience functions
+ self.dev_mode = dev_mode
if node_uid_env is not None:
self.id = UID.from_string(node_uid_env)
else:
if id is None:
id = UID()
self.id = id
- self.packages = get_venv_packages()
+ self.packages = ""
self.signing_key = None
if signing_key_env is not None:
@@ -275,10 +292,14 @@ def __init__(
services = (
[
UserService,
+ WorkerService,
SettingsService,
ActionService,
+ LogService,
DatasetService,
UserCodeService,
+ QueueService,
+ JobService,
RequestService,
DataSubjectService,
NetworkService,
@@ -317,7 +338,7 @@ def __init__(
self._construct_services()
create_admin_new( # nosec B106
- name="Jane Doe",
+ name=root_username,
email=root_email,
password=root_password,
node=self,
@@ -334,16 +355,26 @@ def __init__(
self.post_init()
self.create_initial_settings(admin_email=root_email)
- if not (self.is_subprocess or self.processes == 0):
- self.init_queue_manager(queue_config=queue_config)
+
+ self.init_queue_manager(queue_config=queue_config)
self.init_blob_storage(config=blob_storage_config)
# Migrate data before any operation on db
- self.find_and_migrate_data()
+ if migrate:
+ self.find_and_migrate_data()
NodeRegistry.set_node_for(self.id, self)
+ @property
+ def runs_in_docker(self):
+ path = "/proc/self/cgroup"
+ return (
+ os.path.exists("/.dockerenv")
+ or os.path.isfile(path)
+ and any("docker" in line for line in open(path))
+ )
+
def init_blob_storage(self, config: Optional[BlobStorageConfig] = None) -> None:
if config is None:
root_directory = get_root_data_path()
@@ -355,21 +386,48 @@ def init_blob_storage(self, config: Optional[BlobStorageConfig] = None) -> None:
self.blob_store_config = config_
self.blob_storage_client = config_.client_type(config=config_.client_config)
+ def stop(self):
+ for consumer_list in self.queue_manager.consumers.values():
+ for c in consumer_list:
+ c.close()
+ for p in self.queue_manager.producers.values():
+ p.close()
+
def init_queue_manager(self, queue_config: Optional[QueueConfig]):
queue_config_ = ZMQQueueConfig() if queue_config is None else queue_config
+ self.queue_config = queue_config_
MessageHandlers = [APICallMessageHandler]
self.queue_manager = QueueManager(config=queue_config_)
for message_handler in MessageHandlers:
queue_name = message_handler.queue_name
- producer = self.queue_manager.create_producer(
- queue_name=queue_name,
- )
- consumer = self.queue_manager.create_consumer(
- message_handler, producer.address
- )
- consumer.run()
+ # client config
+ if getattr(queue_config_.client_config, "create_producer", True):
+ context = AuthedServiceContext(
+ node=self,
+ credentials=self.verify_key,
+ role=ServiceRole.ADMIN,
+ )
+ producer: QueueProducer = self.queue_manager.create_producer(
+ queue_name=queue_name, queue_stash=self.queue_stash, context=context
+ )
+ producer.run()
+ address = producer.address
+ else:
+ port = queue_config_.client_config.queue_port
+ if port is not None:
+ address = f"tcp://localhost:{port}"
+ else:
+ address = None
+
+ for _ in range(queue_config_.client_config.n_consumers):
+ if address is None:
+ raise ValueError("address unknown for consumers")
+ consumer: QueueConsumer = self.queue_manager.create_consumer(
+ message_handler, address=address
+ )
+ consumer.run()
@classmethod
def named(
@@ -383,6 +441,11 @@ def named(
node_type: Union[str, NodeType] = NodeType.DOMAIN,
node_side_type: Union[str, NodeSideType] = NodeSideType.HIGH_SIDE,
enable_warnings: bool = False,
+ n_consumers: int = 0,
+ create_producer: bool = False,
+ queue_port: Optional[int] = None,
+ dev_mode: bool = False,
+ migrate: bool = False,
) -> Self:
name_hash = hashlib.sha256(name.encode("utf8")).digest()
name_hash_uuid = name_hash[0:16]
@@ -416,6 +479,12 @@ def named(
db.commit()
db.close()
+ # remove lock files for reading
+ # we should update this to partition locks per node
+ for f in Path("/tmp/sherlock").glob("*.json"): # nosec
+ if f.is_file():
+ f.unlink()
+
with contextlib.suppress(FileNotFoundError, PermissionError):
if os.path.exists(store_config.file_path):
os.unlink(store_config.file_path)
@@ -433,6 +502,17 @@ def named(
client_config=blob_client_config
)
+ if queue_port is not None or n_consumers > 0 or create_producer:
+ queue_config = ZMQQueueConfig(
+ client_config=ZMQClientConfig(
+ create_producer=create_producer,
+ queue_port=queue_port,
+ n_consumers=n_consumers,
+ )
+ )
+ else:
+ queue_config = None
+
return cls(
name=name,
id=uid,
@@ -444,6 +524,9 @@ def named(
node_side_type=node_side_type,
enable_warnings=enable_warnings,
blob_storage_config=blob_storage_config,
+ queue_config=queue_config,
+ dev_mode=dev_mode,
+ migrate=migrate,
)
def is_root(self, credentials: SyftVerifyKey) -> bool:
@@ -669,8 +752,7 @@ def init_stores(
)
elif isinstance(action_store_config, MongoStoreConfig):
self.action_store = MongoActionStore(
- store_config=action_store_config,
- root_verify_key=self.verify_key,
+ root_verify_key=self.verify_key, store_config=action_store_config
)
else:
self.action_store = DictActionStore(root_verify_key=self.verify_key)
@@ -678,6 +760,10 @@ def init_stores(
self.action_store_config = action_store_config
self.queue_stash = QueueStash(store=self.document_store)
+ @property
+ def job_stash(self):
+ return self.get_service("jobservice").stash
+
def _construct_services(self):
self.service_path_map = {}
@@ -687,10 +773,14 @@ def _construct_services(self):
kwargs["store"] = self.action_store
store_services = [
UserService,
+ WorkerService,
SettingsService,
DatasetService,
UserCodeService,
+ LogService,
RequestService,
+ QueueService,
+ JobService,
DataSubjectService,
NetworkService,
PolicyService,
@@ -794,13 +884,37 @@ def __eq__(self, other: Any) -> bool:
return True
+ def await_future(
+ self, credentials: SyftVerifyKey, uid: UID
+ ) -> Union[Optional[QueueItem], SyftError]:
+ # stdlib
+ from time import sleep
+
+ # relative
+ from ..service.queue.queue import Status
+
+ while True:
+ result = self.queue_stash.pop_on_complete(credentials, uid)
+ if not result.is_ok():
+ return result.err()
+ else:
+ res = result.ok()
+ if res.status == Status.COMPLETED:
+ return res
+ sleep(0.1)
+
def resolve_future(
self, credentials: SyftVerifyKey, uid: UID
) -> Union[Optional[QueueItem], SyftError]:
result = self.queue_stash.pop_on_complete(credentials, uid)
if result.is_ok():
- return result.ok()
+ queue_obj = result.ok()
+ queue_obj._set_obj_location_(
+ node_uid=self.id,
+ credentials=credentials,
+ )
+ return queue_obj
return result.err()
def forward_message(
@@ -867,17 +981,25 @@ def get_role_for_credentials(self, credentials: SyftVerifyKey) -> ServiceRole:
return role
def handle_api_call(
- self, api_call: Union[SyftAPICall, SignedSyftAPICall]
+ self,
+ api_call: Union[SyftAPICall, SignedSyftAPICall],
+ job_id: Optional[UID] = None,
+ check_call_location=True,
) -> Result[SignedSyftAPICall, Err]:
# Get the result
- result = self.handle_api_call_with_unsigned_result(api_call)
+ result = self.handle_api_call_with_unsigned_result(
+ api_call, job_id=job_id, check_call_location=check_call_location
+ )
# Sign the result
signed_result = SyftAPIData(data=result).sign(self.signing_key)
return signed_result
def handle_api_call_with_unsigned_result(
- self, api_call: Union[SyftAPICall, SignedSyftAPICall]
+ self,
+ api_call: Union[SyftAPICall, SignedSyftAPICall],
+ job_id: Optional[UID] = None,
+ check_call_location=True,
) -> Result[Union[QueueItem, SyftObject], Err]:
if self.required_signed_calls and isinstance(api_call, SyftAPICall):
return SyftError(
@@ -887,7 +1009,7 @@ def handle_api_call_with_unsigned_result(
if not api_call.is_valid:
return SyftError(message="Your message signature is invalid") # type: ignore
- if api_call.message.node_uid != self.id:
+ if api_call.message.node_uid != self.id and check_call_location:
return self.forward_message(api_call=api_call)
if api_call.message.path == "queue":
return self.resolve_future(
@@ -900,13 +1022,13 @@ def handle_api_call_with_unsigned_result(
result = None
is_blocking = api_call.message.blocking
- if is_blocking or self.is_subprocess or self.processes == 0:
+ if is_blocking or self.is_subprocess:
credentials: SyftVerifyKey = api_call.credentials
api_call = api_call.message
role = self.get_role_for_credentials(credentials=credentials)
context = AuthedServiceContext(
- node=self, credentials=credentials, role=role
+ node=self, credentials=credentials, role=role, job_id=job_id
)
AuthNodeContextRegistry.set_node_context(self.id, context, credentials)
@@ -934,21 +1056,99 @@ def handle_api_call_with_unsigned_result(
message=f"Exception calling {api_call.path}. {traceback.format_exc()}"
)
else:
- task_uid = UID()
- item = QueueItem(id=task_uid, node_uid=self.id)
- # 🟡 TODO 36: Needs distributed lock
- self.queue_stash.set_placeholder(self.verify_key, item)
+ return self.add_api_call_to_queue(api_call)
+ return result
- # Publisher system which pushes to a Queue
- worker_settings = WorkerSettings.from_node(node=self)
+ def add_action_to_queue(
+ self, action, credentials, parent_job_id=None, has_execute_permissions=False
+ ):
+ job_id = UID()
+ task_uid = UID()
+ worker_settings = WorkerSettings.from_node(node=self)
+
+ queue_item = ActionQueueItem(
+ id=task_uid,
+ node_uid=self.id,
+ syft_client_verify_key=credentials,
+ syft_node_location=self.id,
+ job_id=job_id,
+ worker_settings=worker_settings,
+ args=[],
+ kwargs={"action": action},
+ has_execute_permissions=has_execute_permissions,
+ )
+ return self.add_queueitem_to_queue(
+ queue_item, credentials, action, parent_job_id
+ )
- message_bytes = _serialize(
- [task_uid, api_call, worker_settings], to_bytes=True
- )
- self.queue_manager.send(message=message_bytes, queue_name="api_call")
+ def add_queueitem_to_queue(
+ self, queue_item, credentials, action=None, parent_job_id=None
+ ):
+ log_id = UID()
+
+ result_obj = ActionObject.empty()
+ if action is not None:
+ result_obj.id = action.result_id
+ result_obj.syft_resolved = False
+
+ job = Job(
+ id=queue_item.job_id,
+ result=result_obj,
+ node_uid=self.id,
+ syft_client_verify_key=credentials,
+ syft_node_location=self.id,
+ log_id=log_id,
+ parent_job_id=parent_job_id,
+ action=action,
+ )
- return item
- return result
+ # 🟡 TODO 36: Needs distributed lock
+ self.queue_stash.set_placeholder(credentials, queue_item)
+ self.job_stash.set(credentials, job)
+
+ log_service = self.get_service("logservice")
+ role = self.get_role_for_credentials(credentials=credentials)
+ context = AuthedServiceContext(node=self, credentials=credentials, role=role)
+ result = log_service.add(context, log_id)
+ if isinstance(result, SyftError):
+ return result
+ return job
+
+ def add_api_call_to_queue(self, api_call, parent_job_id=None):
+ unsigned_call = api_call
+ if isinstance(api_call, SignedSyftAPICall):
+ unsigned_call = api_call.message
+
+ is_user_code = unsigned_call.path == "code.call"
+
+ service, method = unsigned_call.path.split(".")
+
+ action = None
+ if is_user_code:
+ action = Action.from_api_call(unsigned_call)
+ return self.add_action_to_queue(
+ action, api_call.credentials, parent_job_id=parent_job_id
+ )
+ else:
+ worker_settings = WorkerSettings.from_node(node=self)
+ queue_item = QueueItem(
+ id=UID(),
+ node_uid=self.id,
+ syft_client_verify_key=api_call.credentials,
+ syft_node_location=self.id,
+ job_id=UID(),
+ worker_settings=worker_settings,
+ service=service,
+ method=method,
+ args=unsigned_call.args,
+ kwargs=unsigned_call.kwargs,
+ )
+ return self.add_queueitem_to_queue(
+ queue_item,
+ api_call.credentials,
+ action=None,
+ parent_job_id=parent_job_id,
+ )
def get_api(
self,
@@ -1007,88 +1207,6 @@ def create_initial_settings(self, admin_email: str) -> Optional[NodeSettingsV2]:
print("create_worker_metadata failed", e)
-def task_producer(
- pipe: _GIPCDuplexHandle, api_call: SyftAPICall, blocking: bool
-) -> Any:
- try:
- result = None
- with pipe:
- pipe.put(api_call)
- gevent.sleep(0)
- if blocking:
- try:
- result = pipe.get()
- except EOFError:
- pass
- pipe.close()
- if blocking:
- return result
- except gipc.gipc.GIPCClosed:
- pass
- except Exception as e:
- print("Exception in task_producer", e)
-
-
-def task_runner(
- pipe: _GIPCDuplexHandle,
- worker_settings: WorkerSettings,
- task_uid: UID,
- blocking: bool,
-) -> None:
- worker = Node(
- id=worker_settings.id,
- name=worker_settings.name,
- signing_key=worker_settings.signing_key,
- document_store_config=worker_settings.document_store_config,
- action_store_config=worker_settings.action_store_config,
- blob_storage_config=worker_settings.blob_store_config,
- is_subprocess=True,
- )
- try:
- with pipe:
- api_call = pipe.get()
-
- result = worker.handle_api_call(api_call)
- if blocking:
- pipe.put(result)
- else:
- item = QueueItem(
- node_uid=worker.id, id=task_uid, result=result, resolved=True
- )
- worker.queue_stash.set_result(worker.verify_key, item)
- worker.queue_stash.partition.close()
- pipe.close()
- except Exception as e:
- print("Exception in task_runner", e)
- raise e
-
-
-def queue_task(
- api_call: SyftAPICall,
- worker_settings: WorkerSettings,
- task_uid: UID,
- blocking: bool,
-) -> Optional[Any]:
- with gipc.pipe(encoder=gipc_encoder, decoder=gipc_decoder, duplex=True) as (
- cend,
- pend,
- ):
- process = gipc.start_process(
- task_runner, args=(cend, worker_settings, task_uid, blocking)
- )
- producer = gevent.spawn(task_producer, pend, api_call, blocking)
- try:
- process.join()
- except KeyboardInterrupt:
- producer.kill(block=True)
- process.terminate()
- process.join()
-
- if blocking:
- return producer.value
- return None
-
-
def create_admin_new(
name: str,
email: str,
diff --git a/packages/syft/src/syft/node/routes.py b/packages/syft/src/syft/node/routes.py
index b9f7ffc396d..deeb4fa8c1a 100644
--- a/packages/syft/src/syft/node/routes.py
+++ b/packages/syft/src/syft/node/routes.py
@@ -33,8 +33,12 @@
def make_routes(worker: Worker) -> APIRouter:
if TRACE_MODE:
# third party
- from opentelemetry import trace
- from opentelemetry.propagate import extract
+ try:
+ # third party
+ from opentelemetry import trace
+ from opentelemetry.propagate import extract
+ except Exception:
+ print("Failed to import opentelemetry")
router = APIRouter()
diff --git a/packages/syft/src/syft/node/server.py b/packages/syft/src/syft/node/server.py
index 5d901e4bb23..b78a00d14d1 100644
--- a/packages/syft/src/syft/node/server.py
+++ b/packages/syft/src/syft/node/server.py
@@ -71,6 +71,7 @@ def run_uvicorn(
node_type: Enum,
host: str,
port: int,
+ processes: int,
reset: bool,
dev_mode: bool,
node_side_type: str,
@@ -96,21 +97,23 @@ async def _run_uvicorn(
worker = worker_class.named(
name=name,
- processes=0,
+ processes=processes,
reset=reset,
local_db=True,
node_type=node_type,
node_side_type=node_side_type,
enable_warnings=enable_warnings,
+ migrate=True,
)
else:
worker = worker_class(
name=name,
- processes=0,
+ processes=processes,
local_db=True,
node_type=node_type,
node_side_type=node_side_type,
enable_warnings=enable_warnings,
+ migrate=True,
)
router = make_routes(worker=worker)
app = make_app(worker.name, router=router)
@@ -160,6 +163,7 @@ def serve_node(
node_side_type: NodeSideType = NodeSideType.HIGH_SIDE,
host: str = "0.0.0.0", # nosec
port: int = 8080,
+ processes: int = 1,
reset: bool = False,
dev_mode: bool = False,
tail: bool = False,
@@ -172,6 +176,7 @@ def serve_node(
node_type,
host,
port,
+ processes,
reset,
dev_mode,
node_side_type,
@@ -182,7 +187,11 @@ def serve_node(
def stop():
print(f"Stopping {name}")
server_process.terminate()
- server_process.join()
+ server_process.join(3)
+ if server_process.is_alive():
+ # this is needed because often the process is still alive
+ server_process.kill()
+ print("killed")
def start():
print(f"Starting {name} server on {host}:{port}")
diff --git a/packages/syft/src/syft/node/worker_settings.py b/packages/syft/src/syft/node/worker_settings.py
index 6996fb411ee..106e2e94821 100644
--- a/packages/syft/src/syft/node/worker_settings.py
+++ b/packages/syft/src/syft/node/worker_settings.py
@@ -13,15 +13,20 @@
from ..abstract_node import NodeType
from ..node.credentials import SyftSigningKey
from ..serde.serializable import serializable
+from ..service.queue.base_queue import QueueConfig
from ..store.blob_storage import BlobStorageConfig
from ..store.document_store import StoreConfig
+from ..types.syft_migration import migrate
from ..types.syft_object import SYFT_OBJECT_VERSION_1
+from ..types.syft_object import SYFT_OBJECT_VERSION_2
from ..types.syft_object import SyftObject
+from ..types.transforms import drop
+from ..types.transforms import make_set_default
from ..types.uid import UID
@serializable()
-class WorkerSettings(SyftObject):
+class WorkerSettingsV1(SyftObject):
__canonical_name__ = "WorkerSettings"
__version__ = SYFT_OBJECT_VERSION_1
@@ -34,6 +39,22 @@ class WorkerSettings(SyftObject):
action_store_config: StoreConfig
blob_store_config: Optional[BlobStorageConfig]
+
+@serializable()
+class WorkerSettings(SyftObject):
+ __canonical_name__ = "WorkerSettings"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ id: UID
+ name: str
+ node_type: NodeType
+ node_side_type: NodeSideType
+ signing_key: SyftSigningKey
+ document_store_config: StoreConfig
+ action_store_config: StoreConfig
+ blob_store_config: Optional[BlobStorageConfig]
+ queue_config: Optional[QueueConfig]
+
@staticmethod
def from_node(node: AbstractNode) -> Self:
return WorkerSettings(
@@ -45,4 +66,23 @@ def from_node(node: AbstractNode) -> Self:
action_store_config=node.action_store_config,
node_side_type=node.node_side_type.value,
blob_store_config=node.blob_store_config,
+ queue_config=node.queue_config,
)
+
+
+# queue_config
+
+
+@migrate(WorkerSettings, WorkerSettingsV1)
+def downgrade_workersettings_v2_to_v1():
+ return [
+ drop(["queue_config"]),
+ ]
+
+
+@migrate(WorkerSettingsV1, WorkerSettings)
+def upgrade_workersettings_v1_to_v2():
+ # relative
+ from ..service.queue.zmq_queue import ZMQQueueConfig
+
+ return [make_set_default("queue_config", ZMQQueueConfig())]
diff --git a/packages/syft/src/syft/protocol/data_protocol.py b/packages/syft/src/syft/protocol/data_protocol.py
index 6797c9e8866..700ecd6aeb5 100644
--- a/packages/syft/src/syft/protocol/data_protocol.py
+++ b/packages/syft/src/syft/protocol/data_protocol.py
@@ -79,7 +79,10 @@ def _calculate_object_hash(klass: Type[SyftBaseObject]) -> str:
return hashlib.sha256(json.dumps(obj_meta_info).encode()).hexdigest()
def read_history(self) -> Dict:
- return json.loads(self.file_path.read_text())
+ try:
+ return json.loads(self.file_path.read_text())
+ except Exception:
+ return {}
def save_history(self, history: dict) -> None:
self.file_path.write_text(json.dumps(history, indent=2) + "\n")
@@ -250,7 +253,7 @@ def stage_protocol_changes(self) -> Result[SyftSuccess, SyftError]:
# Sort the version dict
object_versions[canonical_name] = sort_dict_naturally(
- object_versions[canonical_name]
+ object_versions.get(canonical_name, {})
)
current_history["dev"]["object_versions"] = object_versions
diff --git a/packages/syft/src/syft/protocol/protocol_version.json b/packages/syft/src/syft/protocol/protocol_version.json
index 0ea2060243e..e3ba21f633d 100644
--- a/packages/syft/src/syft/protocol/protocol_version.json
+++ b/packages/syft/src/syft/protocol/protocol_version.json
@@ -1,5 +1,5 @@
{
- "1": {
+ "dev": {
"object_versions": {
"PartialSyftObject": {
"1": {
@@ -212,11 +212,73 @@
"action": "add"
}
},
+ "ActionDataEmpty": {
+ "1": {
+ "version": 1,
+ "hash": "89b5912fe5416f922051b8068be6071a03c87a4ab264959de524f1b86e95f028",
+ "action": "add"
+ }
+ },
+ "ActionFileData": {
+ "1": {
+ "version": 1,
+ "hash": "1f32d94b75b0a6b4e86cec93d94aa905738219e3e7e75f51dd335ee832a6ed3e",
+ "action": "add"
+ }
+ },
+ "Action": {
+ "1": {
+ "version": 1,
+ "hash": "5cf71ee35097f17fbb1dd05096f875211d71cf07161205d7f6a9c11fd49d5272",
+ "action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "a13b50c4d23bd6deb7896e394f2a20e6cef4c33c5e6f4ee30f19eaffab708f21",
+ "action": "add"
+ }
+ },
+ "ActionObject": {
+ "1": {
+ "version": 1,
+ "hash": "632446f1415102490c93fafb56dd9eb29d79623bcc5e9f2e6e37c4f63c2c51c3",
+ "action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "577aa1f010b90194958a18ec38ee21db3718bd96d9e036501c6ddeefabedf432",
+ "action": "add"
+ }
+ },
+ "AnyActionObject": {
+ "1": {
+ "version": 1,
+ "hash": "bcb31f847907edc9c95d2d120dc5427854604f40940e3f41cd0474a1820ac65e",
+ "action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "002d8be821140befebbc0503e6bc1ef8779094e24e46305e5da5af6eecb56b13",
+ "action": "add"
+ }
+ },
"BlobFile": {
"1": {
"version": 1,
"hash": "47ed55183d619c6c624e35412360a41de42833e2c24223c1de1ad12a84fdafc2",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "f2b29d28fe81a04bf5e946c819010283a9f98a97d50519358bead773865a2e09",
+ "action": "add"
+ }
+ },
+ "BlobFileOBject": {
+ "1": {
+ "version": 1,
+ "hash": "8da2c80ced4f0414c671313c4b63d05846df1e397c763d99d803be86c29755bb",
+ "action": "add"
}
},
"SecureFilePathLocation": {
@@ -238,6 +300,11 @@
"version": 1,
"hash": "9f1b027cce390ee6f71c7a81e7420bb71a477b29c6c62ba74e781a97bc5434e6",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "5472bdd5bdce6d0b561543a6bac70d47bf0c05c141a21450751460cc538d6b55",
+ "action": "add"
}
},
"BlobStorageMetadata": {
@@ -245,6 +312,11 @@
"version": 1,
"hash": "6888943be3f97186190dd26d7eefbdf29b15c6f2fa459e13608065ebcdb799e2",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "674f4c52a8444289d5ef389b919008860e2b0e7acbaafa774d58e492d5b6741a",
+ "action": "add"
}
},
"CreateBlobStorageEntry": {
@@ -259,6 +331,11 @@
"version": 1,
"hash": "a8d7e1d6483e7a9b5a130e837fa398862aa6cbb316cc5f4470450d835755fdd9",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "4c4fbdb6df5bb9fcbe914a9890bd1c1b6a1b3f382a04cbc8752a5a1b03130111",
+ "action": "add"
}
},
"SyftObjectRetrieval": {
@@ -266,12 +343,17 @@
"version": 1,
"hash": "7ccc62d5b434d2d438b3df661b4d753b0c7c8d593d451d8b86d364da83998c89",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "d9d7a7e1b8843145c9687fd013c9223700285886073547734267e91ac53e0996",
+ "action": "add"
}
},
"BlobRetrievalByURL": {
- "1": {
- "version": 1,
- "hash": "18fd860cb9de296532fc9ff075932e6a4377cc8f043dd88ed4f620517321077d",
+ "2": {
+ "version": 2,
+ "hash": "8059ee03016c4d74e408dad9529e877f91829672e0cc42d8cfff9c8e14058adc",
"action": "add"
}
},
@@ -287,6 +369,42 @@
"version": 1,
"hash": "0dcd95422ec8a7c74e45ee68a125084c08f898dc94a13d25fe5a5fd0e4fc5027",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "d623a8a0d6c83b26ba49686bd8be10eccb126f54626fef334a85396c3b8a8ed6",
+ "action": "add"
+ }
+ },
+ "QueueItem": {
+ "1": {
+ "version": 1,
+ "hash": "5aa94681d9d0715d5b605f9625a54e114927271378cf2ea7245f85c488035e0b",
+ "action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "9503b878de4b5b7a1793580301353523b7d6219ebd27d38abe598061979b7570",
+ "action": "add"
+ }
+ },
+ "ActionQueueItem": {
+ "1": {
+ "version": 1,
+ "hash": "11a43caf9164eb2a5a21f4bcb0ca361d0a5d134bf3c60173f2c502d0d80219de",
+ "action": "add"
+ }
+ },
+ "ZMQClientConfig": {
+ "1": {
+ "version": 1,
+ "hash": "e6054969b495791569caaf33239039beae3d116e1fe74e9575467c48b9007c45",
+ "action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "0f9bc88d56cd6eed6fc75459d1f914aed840c66e1195b9e41cc501b488fef2ed",
+ "action": "add"
}
},
"HTTPNodeRoute": {
@@ -380,38 +498,15 @@
"action": "add"
}
},
- "ActionDataEmpty": {
- "1": {
- "version": 1,
- "hash": "89b5912fe5416f922051b8068be6071a03c87a4ab264959de524f1b86e95f028",
- "action": "add"
- }
- },
- "ActionFileData": {
+ "JobItem": {
"1": {
"version": 1,
- "hash": "1f32d94b75b0a6b4e86cec93d94aa905738219e3e7e75f51dd335ee832a6ed3e",
+ "hash": "7b8723861837b0b7e948b2cf9244159d232185f3407dd6bef108346f941ddf6e",
"action": "add"
- }
- },
- "Action": {
- "1": {
- "version": 1,
- "hash": "5cf71ee35097f17fbb1dd05096f875211d71cf07161205d7f6a9c11fd49d5272",
- "action": "add"
- }
- },
- "ActionObject": {
- "1": {
- "version": 1,
- "hash": "632446f1415102490c93fafb56dd9eb29d79623bcc5e9f2e6e37c4f63c2c51c3",
- "action": "add"
- }
- },
- "AnyActionObject": {
- "1": {
- "version": 1,
- "hash": "bcb31f847907edc9c95d2d120dc5427854604f40940e3f41cd0474a1820ac65e",
+ },
+ "2": {
+ "version": 2,
+ "hash": "e99cf5a78c6dd3a0adc37af3472c7c21570a9e747985dff540a2b06d24de6446",
"action": "add"
}
},
@@ -469,12 +564,17 @@
"version": 1,
"hash": "e14c22686cdc7d1fb2b0d01c0aebdea37e62a61b051677c1d30234214f05cd42",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "660e1abc15034f525e91ffdd820c2a2179bfddf83b7b9e3ce7823b2efc515c69",
+ "action": "add"
}
},
"SubmitUserCode": {
- "1": {
- "version": 1,
- "hash": "f572d32350d09e25b29572c591029d37a216818618c383094404f84bc9c15dd6",
+ "2": {
+ "version": 2,
+ "hash": "9b29e060973a3de8d3564a2b7d2bb5c53745aa445bf257576994b613505d7194",
"action": "add"
}
},
@@ -546,6 +646,11 @@
"version": 1,
"hash": "dcc7b44fa5ad22ae0bc576948f856c172dac1e9de2bc8e2a302e428f3309a278",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "2c631121d9211006edab5620b214dea83e2398bee92244d822227ee316647e22",
+ "action": "add"
}
},
"NumpyScalarObject": {
@@ -553,6 +658,11 @@
"version": 1,
"hash": "5c1b6b6e8ba88bc79e76646d621489b889fe8f9b9fd59f117d594be18a409633",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "0d5d81b9d45c140f6e07b43ed68d31e0ef060d6b4d0431c9b4795997bb35c69d",
+ "action": "add"
}
},
"NumpyBoolObject": {
@@ -560,6 +670,11 @@
"version": 1,
"hash": "a5c822a6a3ca9eefd6a2b68f7fd0bc614fba7995f6bcc30bdc9dc882296b9b16",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "24839ba1c88ed833a134124750d5f299abcdf318670315028ed87b254f4578b3",
+ "action": "add"
}
},
"PandasDataframeObject": {
@@ -567,6 +682,11 @@
"version": 1,
"hash": "35058924b3de2e0a604a92f91f4dd2e3cc0dac80c219d34f360e7cedd52f5f4c",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "66729d4ba7a92210d45c5a5c24fbdb4c8e58138a515a7bdb71ac8f6e8b868544",
+ "action": "add"
}
},
"PandasSeriesObject": {
@@ -574,6 +694,11 @@
"version": 1,
"hash": "2a0d8a55f1c27bd8fccd276cbe01bf272c40cab10417d7027273983fed423caa",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "cb05a714f75b1140a943f56a3622fcc0477b3a1f504cd545a98510959ffe1528",
+ "action": "add"
}
},
"ReplyNotification": {
@@ -665,6 +790,23 @@
"version": 1,
"hash": "4f5b405cc2b3976ed8f7018df82e873435d9187dff15fa5a23bc85a738969f3f",
"action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "d83e0905ae882c824ba8fbbf455cd3881906bf8b2ebbfff07bcf471ef869cedc",
+ "action": "add"
+ }
+ },
+ "SyftLog": {
+ "1": {
+ "version": 1,
+ "hash": "bd3f62b8fe4b2718a6380c8f05a93c5c40169fc4ab174db291929298e588429e",
+ "action": "add"
+ },
+ "2": {
+ "version": 2,
+ "hash": "d3ce45794da2e6c4b0cef63b98a553525af50c5d9db42d3d64caef3e7d22b4a9",
+ "action": "add"
}
},
"SyftObjectMigrationState": {
@@ -730,17 +872,10 @@
"action": "add"
}
},
- "QueueItem": {
+ "ContainerImage": {
"1": {
"version": 1,
- "hash": "5aa94681d9d0715d5b605f9625a54e114927271378cf2ea7245f85c488035e0b",
- "action": "add"
- }
- },
- "ZMQClientConfig": {
- "1": {
- "version": 1,
- "hash": "e6054969b495791569caaf33239039beae3d116e1fe74e9575467c48b9007c45",
+ "hash": "776fc7cf7498b93e656a00fff03b86160d1b63e508e2143ac7932e7e38021b0c",
"action": "add"
}
},
diff --git a/packages/syft/src/syft/service/action/action_data_empty.py b/packages/syft/src/syft/service/action/action_data_empty.py
index d1e5ae44381..6183d4153a8 100644
--- a/packages/syft/src/syft/service/action/action_data_empty.py
+++ b/packages/syft/src/syft/service/action/action_data_empty.py
@@ -26,10 +26,10 @@ class ActionDataEmpty(SyftObject):
syft_internal_type: Optional[Type] = NoneType
def __repr__(self) -> str:
- return f"{type(self).__name__} UID: {self.id} <{self.syft_internal_type}>"
+ return f"{type(self).__name__} <{self.syft_internal_type}>"
def __str__(self) -> str:
- return f"{type(self).__name__} UID: {self.id} <{self.syft_internal_type}>"
+ return f"{type(self).__name__} <{self.syft_internal_type}>"
@serializable()
@@ -47,4 +47,5 @@ def __validate_file_path(cls, v: Union[str, Path]) -> Path:
if v.exists() and v.is_file():
return v
- raise ValueError(f"Not a valid path to file. {v}")
+ # this breaks server side during deserialization
+ # raise ValueError(f"Not a valid path to file. {v}")
diff --git a/packages/syft/src/syft/service/action/action_object.py b/packages/syft/src/syft/service/action/action_object.py
index 94ef6f66d12..ed6593a014d 100644
--- a/packages/syft/src/syft/service/action/action_object.py
+++ b/packages/syft/src/syft/service/action/action_object.py
@@ -26,19 +26,23 @@
from typing_extensions import Self
# relative
+from ...client.api import APIRegistry
from ...client.api import SyftAPI
+from ...client.api import SyftAPICall
from ...client.client import SyftClient
from ...node.credentials import SyftVerifyKey
from ...serde.serializable import serializable
from ...serde.serialize import _serialize as serialize
from ...service.response import SyftError
-from ...store.blob_storage import BlobRetrieval
from ...store.linked_obj import LinkedObject
-from ...types.blob_storage import CreateBlobStorageEntry
from ...types.datetime import DateTime
+from ...types.syft_migration import migrate
from ...types.syft_object import SYFT_OBJECT_VERSION_1
+from ...types.syft_object import SYFT_OBJECT_VERSION_2
from ...types.syft_object import SyftBaseObject
from ...types.syft_object import SyftObject
+from ...types.transforms import drop
+from ...types.transforms import make_set_default
from ...types.uid import LineageID
from ...types.uid import UID
from ...util.logger import debug
@@ -68,6 +72,7 @@ class ActionType(Enum):
SETATTRIBUTE = 4
FUNCTION = 8
CREATEOBJECT = 16
+ SYFTFUNCTION = 32
def repr_cls(c):
@@ -75,7 +80,7 @@ def repr_cls(c):
@serializable()
-class Action(SyftObject):
+class ActionV1(SyftObject):
"""Serializable Action object.
Parameters:
@@ -107,6 +112,41 @@ class Action(SyftObject):
action_type: Optional[ActionType]
create_object: Optional[SyftObject] = None
+
+@serializable()
+class Action(SyftObject):
+ """Serializable Action object.
+
+ Parameters:
+ path: str
+ The path of the Type of the remote object.
+ op: str
+ The method to be executed from the remote object.
+ remote_self: Optional[LineageID]
+ The extended UID of the SyftObject
+ args: List[LineageID]
+ `op` args
+ kwargs: Dict[str, LineageID]
+ `op` kwargs
+ result_id: Optional[LineageID]
+ Extended UID of the resulted SyftObject
+ """
+
+ __canonical_name__ = "Action"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ __attr_searchable__: List[str] = []
+
+ path: Optional[str]
+ op: Optional[str]
+ remote_self: Optional[LineageID]
+ args: List[LineageID]
+ kwargs: Dict[str, LineageID]
+ result_id: Optional[LineageID]
+ action_type: Optional[ActionType]
+ create_object: Optional[SyftObject] = None
+ user_code_id: Optional[UID] = None
+
@pydantic.validator("id", pre=True, always=True)
def make_id(cls, v: Optional[UID]) -> UID:
"""Generate or reuse an UID"""
@@ -122,6 +162,18 @@ def full_path(self) -> str:
"""Action path and operation"""
return f"{self.path}.{self.op}"
+ @property
+ def job_display_name(self) -> str:
+ if self.user_code_id is not None:
+ api = APIRegistry.api_for(
+ node_uid=self.syft_node_location,
+ user_verify_key=self.syft_client_verify_key,
+ )
+ user_code = api.services.code.get_by_id(self.user_code_id)
+ return user_code.service_func_name
+ else:
+ return f"{self.path}.{self.op}"
+
@property
def syft_history_hash(self) -> int:
"""Create a unique hash for the operations applied on the object."""
@@ -141,6 +193,41 @@ def syft_history_hash(self) -> int:
hashes += hash(arg.syft_history_hash)
return hashes
+ @classmethod
+ def syft_function_action_from_kwargs_and_id(cls, kwargs, user_code_id):
+ kwarg_ids = {}
+ for k, v in kwargs.items():
+ kwarg_ids[k] = LineageID(v)
+ return cls(
+ args=[],
+ kwargs=kwarg_ids,
+ result_id=LineageID(),
+ action_type=ActionType.SYFTFUNCTION,
+ user_code_id=user_code_id,
+ )
+
+ @classmethod
+ def from_api_call(cls, api_call: SyftAPICall) -> Action:
+ # relative
+ from ..code.user_code_service import map_kwargs_to_id
+
+ kwargs = api_call.kwargs
+ kwargs.pop("communication_protocol", None)
+ function_id = kwargs.pop("uid", None)
+ kwargs = map_kwargs_to_id(kwargs)
+ kwarg_ids = {}
+ for k, v in kwargs.items():
+ kwarg_ids[k] = LineageID(v)
+
+ action = cls(
+ args=[],
+ kwargs=kwarg_ids,
+ result_id=LineageID(),
+ action_type=ActionType.SYFTFUNCTION,
+ user_code_id=function_id,
+ )
+ return action
+
def __repr__(self):
def repr_uid(_id):
return f"{str(_id)[:3]}..{str(_id)[-1]}"
@@ -157,6 +244,20 @@ def repr_uid(_id):
)
+@migrate(Action, ActionV1)
+def downgrade_action_v2_to_v1():
+ return [
+ drop("user_code_id"),
+ make_set_default("op", ""),
+ make_set_default("path", ""),
+ ]
+
+
+@migrate(ActionV1, Action)
+def upgrade_action_v1_to_v2():
+ return [make_set_default("user_code_id", None)]
+
+
class ActionObjectPointer:
pass
@@ -193,6 +294,15 @@ class ActionObjectPointer:
"delete_data", # syft
"_save_to_blob_storage_", # syft
"syft_action_data", # syft
+ "syft_resolved", # syft
+ "migrate_to", # syft
+ "to_dict", # syft
+ "dict", # syft
+ "_iter", # pydantic
+ "__exclude_fields__", # pydantic
+ "__include_fields__", # pydantic
+ "_calculate_keys", # pydantic
+ "_get_value", # pydantic
]
dont_wrap_output_attrs = [
"__repr__",
@@ -205,6 +315,7 @@ class ActionObjectPointer:
"__array_wrap__",
"__bool__",
"__len__",
+ "syft_resolved", # syft
]
dont_make_side_effects = [
"_repr_html_",
@@ -215,6 +326,7 @@ class ActionObjectPointer:
"__setitem__",
"__len__",
"shape",
+ "syft_resolved", # syft
]
action_data_empty_must_run = [
"__repr__",
@@ -447,11 +559,12 @@ def debox_args_and_kwargs(args: Any, kwargs: Any) -> Tuple[Any, Any]:
"_set_obj_location_",
"syft_action_data_cache",
"reload_cache",
+ "syft_resolved",
]
@serializable()
-class ActionObject(SyftObject):
+class ActionObjectV1(SyftObject):
"""Action object for remote execution."""
__canonical_name__ = "ActionObject"
@@ -480,6 +593,39 @@ class ActionObject(SyftObject):
syft_has_bool_attr: Optional[bool]
syft_resolve_data: Optional[bool]
syft_created_at: Optional[DateTime]
+
+
+@serializable()
+class ActionObject(SyftObject):
+ """Action object for remote execution."""
+
+ __canonical_name__ = "ActionObject"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ __attr_searchable__: List[str] = []
+ syft_action_data_cache: Optional[Any] = None
+ syft_blob_storage_entry_id: Optional[UID] = None
+ syft_pointer_type: ClassVar[Type[ActionObjectPointer]]
+
+ # Help with calculating history hash for code verification
+ syft_parent_hashes: Optional[Union[int, List[int]]]
+ syft_parent_op: Optional[str]
+ syft_parent_args: Optional[Any]
+ syft_parent_kwargs: Optional[Any]
+ syft_history_hash: Optional[int]
+ syft_internal_type: ClassVar[Type[Any]]
+ syft_node_uid: Optional[UID]
+ _syft_pre_hooks__: Dict[str, List] = {}
+ _syft_post_hooks__: Dict[str, List] = {}
+ syft_twin_type: TwinMode = TwinMode.NONE
+ syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
+ syft_action_data_type: Optional[Type]
+ syft_action_data_repr_: Optional[str]
+ syft_action_data_str_: Optional[str]
+ syft_has_bool_attr: Optional[bool]
+ syft_resolve_data: Optional[bool]
+ syft_created_at: Optional[DateTime]
+ syft_resolved: bool = True
# syft_dont_wrap_attrs = ["shape"]
@property
@@ -512,12 +658,13 @@ def reload_cache(self):
blob_retrieval_object,
)
return blob_retrieval_object
+ # relative
+ from ...store.blob_storage import BlobRetrieval
if isinstance(blob_retrieval_object, SyftError):
raise SyftException(
message=f"Failed to retrieve object from blob storage: {blob_retrieval_object.message}"
)
-
elif isinstance(blob_retrieval_object, BlobRetrieval):
# TODO: This change is temporary to for gateway to be compatible with the new blob storage
self.syft_action_data_cache = blob_retrieval_object.read()
@@ -528,8 +675,13 @@ def reload_cache(self):
# Currently , we are just passing the object as it is, which would be fixed later.
self.syft_action_data_cache = blob_retrieval_object
self.syft_action_data_type = type(self.syft_action_data)
+ else:
+ print("cannot reload cache")
def _save_to_blob_storage_(self, data: Any) -> None:
+ # relative
+ from ...types.blob_storage import CreateBlobStorageEntry
+
if not isinstance(data, ActionDataEmpty):
if isinstance(data, ActionFileData):
storage_entry = CreateBlobStorageEntry.from_path(data.path)
@@ -560,6 +712,8 @@ def _save_to_blob_storage_(self, data: Any) -> None:
self.syft_blob_storage_entry_id = (
blob_deposit_object.blob_storage_entry_id
)
+ else:
+ print("cannot save to blob storage")
self.syft_action_data_type = type(data)
@@ -583,7 +737,7 @@ def _save_to_blob_storage(self) -> Optional[SyftError]:
if isinstance(data, SyftError):
return data
if isinstance(data, ActionDataEmpty):
- return SyftError(f"cannot store empty object {self.id}")
+ return SyftError(message=f"cannot store empty object {self.id}")
result = self._save_to_blob_storage_(data)
if isinstance(result, SyftError):
return result
@@ -944,14 +1098,20 @@ def get(self) -> Any:
if not isinstance(res, ActionObject):
return SyftError(message=f"{res}")
else:
- return res.syft_action_data
+ nested_res = res.syft_action_data
+ if isinstance(nested_res, ActionObject):
+ nested_res.syft_node_location = res.syft_node_location
+ nested_res.syft_client_verify_key = res.syft_client_verify_key
+ return nested_res
def as_empty(self):
id = self.id
# TODO: fix
if isinstance(id, LineageID):
id = id.id
- return ActionObject.empty(self.syft_internal_type, id, self.syft_lineage_id)
+ return ActionObject.empty(
+ self.syft_internal_type, id, self.syft_lineage_id, self.syft_resolved
+ )
@staticmethod
def from_path(
@@ -997,6 +1157,7 @@ def from_obj(
syft_lineage_id: Optional[LineageID] = None,
syft_client_verify_key: Optional[SyftVerifyKey] = None,
syft_node_location: Optional[UID] = None,
+ syft_resolved: Optional[bool] = True,
) -> ActionObject:
"""Create an ActionObject from an existing object.
@@ -1013,6 +1174,7 @@ def from_obj(
action_type = action_type_for_object(syft_action_data)
action_object = action_type(syft_action_data_cache=syft_action_data)
+ action_object.syft_resolved = syft_resolved
if id is not None:
action_object.id = id
@@ -1050,6 +1212,7 @@ def empty(
syft_internal_type: Type[Any] = NoneType,
id: Optional[UID] = None,
syft_lineage_id: Optional[LineageID] = None,
+ syft_resolved: Optional[bool] = True,
) -> ActionObject:
"""Create an ActionObject from a type, using a ActionDataEmpty object
@@ -1064,7 +1227,10 @@ def empty(
empty = ActionDataEmpty(syft_internal_type=syft_internal_type)
res = ActionObject.from_obj(
- id=id, syft_lineage_id=syft_lineage_id, syft_action_data=empty
+ id=id,
+ syft_lineage_id=syft_lineage_id,
+ syft_action_data=empty,
+ syft_resolved=syft_resolved,
)
res.__dict__["syft_internal_type"] = syft_internal_type
return res
@@ -1517,7 +1683,7 @@ def __call__(self, *args: Any, **kwds: Any) -> Any:
return self.__call__(*args, **kwds)
def __str__(self) -> str:
- if not inspect.isclass:
+ if not inspect.isclass(self):
return self.__str__()
else:
return self.syft_action_data_str_
@@ -1669,14 +1835,39 @@ def __rrshift__(self, other: Any) -> Any:
return self._syft_output_action_object(self.__rrshift__(other))
+@migrate(ActionObject, ActionObjectV1)
+def downgrade_actionobject_v2_to_v1():
+ return [
+ drop("syft_resolved"),
+ ]
+
+
+@migrate(ActionObjectV1, ActionObject)
+def upgrade_actionobject_v1_to_v2():
+ return [
+ make_set_default("syft_resolved", True),
+ ]
+
+
@serializable()
-class AnyActionObject(ActionObject):
+class AnyActionObjectV1(ActionObjectV1):
__canonical_name__ = "AnyActionObject"
__version__ = SYFT_OBJECT_VERSION_1
syft_internal_type: ClassVar[Type[Any]] = NoneType # type: ignore
# syft_passthrough_attrs: List[str] = []
- syft_dont_wrap_attrs: List[str] = ["__str__", "__repr__"]
+ syft_dont_wrap_attrs: List[str] = ["__str__", "__repr__", "syft_action_data_str_"]
+
+
+@serializable()
+class AnyActionObject(ActionObject):
+ __canonical_name__ = "AnyActionObject"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ syft_internal_type: ClassVar[Type[Any]] = NoneType # type: ignore
+ # syft_passthrough_attrs: List[str] = []
+ syft_dont_wrap_attrs: List[str] = ["__str__", "__repr__", "syft_action_data_str_"]
+ syft_action_data_str_ = ""
def __float__(self) -> float:
return float(self.syft_action_data)
@@ -1685,6 +1876,22 @@ def __int__(self) -> float:
return int(self.syft_action_data)
+@migrate(AnyActionObject, AnyActionObjectV1)
+def downgrade_anyactionobject_v2_to_v1():
+ return [
+ drop("syft_action_data_str"),
+ drop("syft_resolved"),
+ ]
+
+
+@migrate(AnyActionObjectV1, AnyActionObject)
+def upgrade_anyactionobject_v1_to_v2():
+ return [
+ make_set_default("syft_action_data_str", ""),
+ make_set_default("syft_resolved", True),
+ ]
+
+
action_types[Any] = AnyActionObject
diff --git a/packages/syft/src/syft/service/action/action_service.py b/packages/syft/src/syft/service/action/action_service.py
index 80db6428211..bc30642c7d1 100644
--- a/packages/syft/src/syft/service/action/action_service.py
+++ b/packages/syft/src/syft/service/action/action_service.py
@@ -3,6 +3,7 @@
from typing import Any
from typing import Dict
from typing import List
+from typing import Optional
from typing import Union
# third party
@@ -21,6 +22,7 @@
from ..code.user_code import UserCode
from ..code.user_code import execute_byte_code
from ..context import AuthedServiceContext
+from ..policy.policy import retrieve_from_db
from ..response import SyftError
from ..response import SyftSuccess
from ..service import AbstractService
@@ -180,24 +182,40 @@ def _user_code_execute(
context: AuthedServiceContext,
code_item: UserCode,
kwargs: Dict[str, Any],
+ result_id: Optional[UID] = None,
) -> Result[ActionObjectPointer, Err]:
- filtered_kwargs = code_item.input_policy.filter_kwargs(
- kwargs=kwargs, context=context, code_item_id=code_item.id
- )
-
- if filtered_kwargs.is_err():
- return filtered_kwargs
- filtered_kwargs = filtered_kwargs.ok()
-
- expected_input_kwargs = set()
- for _inp_kwarg in code_item.input_policy.inputs.values():
- expected_input_kwargs.update(_inp_kwarg.keys())
- permitted_input_kwargs = list(filtered_kwargs.keys())
- not_approved_kwargs = set(expected_input_kwargs) - set(permitted_input_kwargs)
- if len(not_approved_kwargs) > 0:
- return Err(
- f"Input arguments: {not_approved_kwargs} to the function are not approved yet."
+ if not context.has_execute_permissions:
+ input_policy = code_item.input_policy
+ filtered_kwargs = input_policy.filter_kwargs(
+ kwargs=kwargs, context=context, code_item_id=code_item.id
)
+ if isinstance(filtered_kwargs, SyftError) or filtered_kwargs.is_err():
+ return filtered_kwargs
+ filtered_kwargs = filtered_kwargs.ok()
+ else:
+ filtered_kwargs = retrieve_from_db(code_item.id, kwargs, context).ok()
+ # update input policy to track any input state
+ # code_item.input_policy = input_policy
+
+ if not context.has_execute_permissions:
+ expected_input_kwargs = set()
+ for _inp_kwarg in code_item.input_policy.inputs.values():
+ keys = _inp_kwarg.keys()
+ for k in keys:
+ if k not in kwargs:
+ return Err(
+ f"{code_item.service_func_name}() missing required keyword argument: '{k}'"
+ )
+ expected_input_kwargs.update(keys)
+
+ permitted_input_kwargs = list(filtered_kwargs.keys())
+ not_approved_kwargs = set(expected_input_kwargs) - set(
+ permitted_input_kwargs
+ )
+ if len(not_approved_kwargs) > 0:
+ return Err(
+ f"Input arguments: {not_approved_kwargs} to the function are not approved yet."
+ )
has_twin_inputs = False
@@ -207,7 +225,7 @@ def _user_code_execute(
has_twin_inputs = True
real_kwargs[key] = kwarg_value
- result_id = UID()
+ result_id = UID() if result_id is None else result_id
try:
if not has_twin_inputs:
@@ -215,23 +233,33 @@ def _user_code_execute(
filtered_kwargs = filter_twin_kwargs(
real_kwargs, twin_mode=TwinMode.NONE
)
- exec_result = execute_byte_code(code_item, filtered_kwargs)
+ exec_result = execute_byte_code(code_item, filtered_kwargs, context)
result_action_object = wrap_result(result_id, exec_result.result)
else:
# twins
private_kwargs = filter_twin_kwargs(
real_kwargs, twin_mode=TwinMode.PRIVATE
)
- private_exec_result = execute_byte_code(code_item, private_kwargs)
+ private_exec_result = execute_byte_code(
+ code_item, private_kwargs, context
+ )
result_action_object_private = wrap_result(
result_id, private_exec_result.result
)
mock_kwargs = filter_twin_kwargs(real_kwargs, twin_mode=TwinMode.MOCK)
- mock_exec_result = execute_byte_code(code_item, mock_kwargs)
- result_action_object_mock = wrap_result(
- result_id, mock_exec_result.result
- )
+ # relative
+ from .action_data_empty import ActionDataEmpty
+
+ if any(isinstance(v, ActionDataEmpty) for v in mock_kwargs.values()):
+ mock_exec_result_obj = ActionDataEmpty()
+ else:
+ mock_exec_result = execute_byte_code(
+ code_item, mock_kwargs, context
+ )
+ mock_exec_result_obj = mock_exec_result.result
+
+ result_action_object_mock = wrap_result(result_id, mock_exec_result_obj)
result_action_object = TwinObject(
id=result_id,
@@ -239,7 +267,18 @@ def _user_code_execute(
mock_obj=result_action_object_mock,
)
except Exception as e:
+ # import traceback
+ # return Err(f"_user_code_execute failed. {e} {traceback.format_exc()}")
return Err(f"_user_code_execute failed. {e}")
+ return Ok(result_action_object)
+
+ def set_result_to_store(self, result_action_object, context, output_policy):
+ result_id = result_action_object.id
+ # result_blob_id = result_action_object.syft_blob_storage_entry_id
+ output_readers = (
+ output_policy.output_readers if not context.has_execute_permissions else []
+ )
+ read_permission = ActionPermission.READ
result_action_object._set_obj_location_(
context.node.id,
@@ -249,35 +288,36 @@ def _user_code_execute(
if isinstance(blob_store_result, SyftError):
return blob_store_result
+ # IMPORTANT: DO THIS ONLY AFTER ._save_to_blob_storage
+ if isinstance(result_action_object, TwinObject):
+ result_blob_id = result_action_object.private.syft_blob_storage_entry_id
+ else:
+ result_blob_id = result_action_object.syft_blob_storage_entry_id
+
# pass permission information to the action store as extra kwargs
context.extra_kwargs = {"has_result_read_permission": True}
set_result = self.set(context, result_action_object)
if set_result.is_err():
- return set_result.err()
+ return set_result
blob_storage_service: BlobStorageService = context.node.get_service(
BlobStorageService
)
- if len(code_item.output_policy.output_readers) > 0:
- self.store.add_permissions(
- [
- ActionObjectPermission(result_id, ActionPermission.READ, x)
- for x in code_item.output_policy.output_readers
- ]
- )
- blob_storage_service.stash.add_permissions(
- [
- ActionObjectPermission(
- result_action_object.syft_blob_storage_entry_id,
- ActionPermission.READ,
- x,
- )
- for x in code_item.output_policy.output_readers
- ]
- )
+ def store_permission(x):
+ return ActionObjectPermission(result_id, read_permission, x)
+
+ def blob_permission(x):
+ return ActionObjectPermission(result_blob_id, read_permission, x)
+
+ if len(output_readers) > 0:
+ store_permissions = [store_permission(x) for x in output_readers]
+ self.store.add_permissions(store_permissions)
+
+ blob_permissions = [blob_permission(x) for x in output_readers]
+ blob_storage_service.stash.add_permissions(blob_permissions)
return set_result
@@ -449,6 +489,16 @@ def execute(
if action.action_type == ActionType.CREATEOBJECT:
result_action_object = Ok(action.create_object)
# print(action.create_object, "already in blob storage")
+ elif action.action_type == ActionType.SYFTFUNCTION:
+ usercode_service = context.node.get_service("usercodeservice")
+ kwarg_ids = {}
+ for k, v in action.kwargs.items():
+ # transform lineage ids into ids
+ kwarg_ids[k] = v.id
+ result_action_object: Result[ActionObject, Err] = usercode_service._call(
+ context, action.user_code_id, action.result_id, **kwarg_ids
+ )
+ return result_action_object
elif action.action_type == ActionType.FUNCTION:
result_action_object = self.call_function(context, action)
else:
diff --git a/packages/syft/src/syft/service/action/action_store.py b/packages/syft/src/syft/service/action/action_store.py
index 25b510cb8ff..b939de6aada 100644
--- a/packages/syft/src/syft/service/action/action_store.py
+++ b/packages/syft/src/syft/service/action/action_store.py
@@ -2,6 +2,7 @@
from __future__ import annotations
# stdlib
+import threading
from typing import List
from typing import Optional
@@ -30,6 +31,8 @@
from .action_permissions import ActionObjectWRITE
from .action_permissions import ActionPermission
+lock = threading.RLock()
+
class ActionStore:
pass
@@ -251,13 +254,15 @@ def migrate_data(self, to_klass: SyftObject, credentials: SyftVerifyKey):
has_root_permission = credentials == self.root_verify_key
if has_root_permission:
- for key, value in self.data:
+ for key, value in self.data.items():
try:
if value.__canonical_name__ != to_klass.__canonical_name__:
continue
- migrated_value = value.migrate_to(to_klass)
- except Exception:
- return Err(f"Failed to migrate data to {to_klass} for qk: {key}")
+ migrated_value = value.migrate_to(to_klass.__version__)
+ except Exception as e:
+ return Err(
+ f"Failed to migrate data to {to_klass} for qk: {key}. Exception: {e}"
+ )
result = self.set(
uid=key,
credentials=credentials,
diff --git a/packages/syft/src/syft/service/action/numpy.py b/packages/syft/src/syft/service/action/numpy.py
index 3c19aa61bc2..45c778b58ab 100644
--- a/packages/syft/src/syft/service/action/numpy.py
+++ b/packages/syft/src/syft/service/action/numpy.py
@@ -8,8 +8,13 @@
# relative
from ...serde.serializable import serializable
+from ...types.syft_migration import migrate
from ...types.syft_object import SYFT_OBJECT_VERSION_1
+from ...types.syft_object import SYFT_OBJECT_VERSION_2
+from ...types.transforms import drop
+from ...types.transforms import make_set_default
from .action_object import ActionObject
+from .action_object import ActionObjectV1
from .action_object import BASE_PASSTHROUGH_ATTRS
from .action_types import action_types
@@ -37,12 +42,23 @@ def numpy_like_eq(left: Any, right: Any) -> bool:
return bool(result)
+@serializable()
+class NumpyArrayObjectV1(ActionObjectV1, np.lib.mixins.NDArrayOperatorsMixin):
+ __canonical_name__ = "NumpyArrayObject"
+ __version__ = SYFT_OBJECT_VERSION_1
+
+ syft_internal_type: ClassVar[Type[Any]] = np.ndarray
+ syft_pointer_type = NumpyArrayObjectPointer
+ syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
+ syft_dont_wrap_attrs = ["dtype", "shape"]
+
+
# 🔵 TODO 7: Map TPActionObjects and their 3rd Party types like numpy type to these
# classes for bi-directional lookup.
@serializable()
class NumpyArrayObject(ActionObject, np.lib.mixins.NDArrayOperatorsMixin):
__canonical_name__ = "NumpyArrayObject"
- __version__ = SYFT_OBJECT_VERSION_1
+ __version__ = SYFT_OBJECT_VERSION_2
syft_internal_type: ClassVar[Type[Any]] = np.ndarray
syft_pointer_type = NumpyArrayObjectPointer
@@ -78,8 +94,22 @@ def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
)
+@migrate(NumpyArrayObject, NumpyArrayObjectV1)
+def downgrade_numpyarrayobject_v2_to_v1():
+ return [
+ drop("syft_resolved"),
+ ]
+
+
+@migrate(NumpyArrayObjectV1, NumpyArrayObject)
+def upgrade_numpyarrayobject_v1_to_v2():
+ return [
+ make_set_default("syft_resolved", True),
+ ]
+
+
@serializable()
-class NumpyScalarObject(ActionObject, np.lib.mixins.NDArrayOperatorsMixin):
+class NumpyScalarObjectV1(ActionObjectV1, np.lib.mixins.NDArrayOperatorsMixin):
__canonical_name__ = "NumpyScalarObject"
__version__ = SYFT_OBJECT_VERSION_1
@@ -87,12 +117,36 @@ class NumpyScalarObject(ActionObject, np.lib.mixins.NDArrayOperatorsMixin):
syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
syft_dont_wrap_attrs = ["dtype", "shape"]
+
+@serializable()
+class NumpyScalarObject(ActionObject, np.lib.mixins.NDArrayOperatorsMixin):
+ __canonical_name__ = "NumpyScalarObject"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ syft_internal_type = np.number
+ syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
+ syft_dont_wrap_attrs = ["dtype", "shape"]
+
def __float__(self) -> float:
return float(self.syft_action_data)
+@migrate(NumpyScalarObject, NumpyScalarObjectV1)
+def downgrade_numpyscalarobject_v2_to_v1():
+ return [
+ drop("syft_resolved"),
+ ]
+
+
+@migrate(NumpyScalarObjectV1, NumpyScalarObject)
+def upgrade_numpyscalarobject_v1_to_v2():
+ return [
+ make_set_default("syft_resolved", True),
+ ]
+
+
@serializable()
-class NumpyBoolObject(ActionObject, np.lib.mixins.NDArrayOperatorsMixin):
+class NumpyBoolObjectV1(ActionObjectV1, np.lib.mixins.NDArrayOperatorsMixin):
__canonical_name__ = "NumpyBoolObject"
__version__ = SYFT_OBJECT_VERSION_1
@@ -101,6 +155,30 @@ class NumpyBoolObject(ActionObject, np.lib.mixins.NDArrayOperatorsMixin):
syft_dont_wrap_attrs = ["dtype", "shape"]
+@serializable()
+class NumpyBoolObject(ActionObject, np.lib.mixins.NDArrayOperatorsMixin):
+ __canonical_name__ = "NumpyBoolObject"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ syft_internal_type = np.bool_
+ syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
+ syft_dont_wrap_attrs = ["dtype", "shape"]
+
+
+@migrate(NumpyBoolObject, NumpyBoolObjectV1)
+def downgrade_numpyboolobject_v2_to_v1():
+ return [
+ drop("syft_resolved"),
+ ]
+
+
+@migrate(NumpyBoolObjectV1, NumpyBoolObject)
+def upgrade_numpyboolobject_v1_to_v2():
+ return [
+ make_set_default("syft_resolved", True),
+ ]
+
+
np_array = np.array([1, 2, 3])
action_types[type(np_array)] = NumpyArrayObject
diff --git a/packages/syft/src/syft/service/action/pandas.py b/packages/syft/src/syft/service/action/pandas.py
index cd669ff1425..a466545b363 100644
--- a/packages/syft/src/syft/service/action/pandas.py
+++ b/packages/syft/src/syft/service/action/pandas.py
@@ -9,19 +9,33 @@
# relative
from ...serde.serializable import serializable
+from ...types.syft_migration import migrate
from ...types.syft_object import SYFT_OBJECT_VERSION_1
+from ...types.syft_object import SYFT_OBJECT_VERSION_2
+from ...types.transforms import drop
+from ...types.transforms import make_set_default
from .action_object import ActionObject
+from .action_object import ActionObjectV1
from .action_object import BASE_PASSTHROUGH_ATTRS
from .action_types import action_types
@serializable()
-class PandasDataFrameObject(ActionObject):
+class PandasDataFrameObjectV1(ActionObjectV1):
__canonical_name__ = "PandasDataframeObject"
__version__ = SYFT_OBJECT_VERSION_1
syft_internal_type: ClassVar[Type[Any]] = DataFrame
syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
+
+
+@serializable()
+class PandasDataFrameObject(ActionObject):
+ __canonical_name__ = "PandasDataframeObject"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ syft_internal_type: ClassVar[Type[Any]] = DataFrame
+ syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
# this is added for instance checks for dataframes
# syft_dont_wrap_attrs = ["shape"]
@@ -41,14 +55,37 @@ def syft_is_property(self, obj: Any, method: str) -> bool:
return super().syft_is_property(obj, method)
+@migrate(PandasDataFrameObject, PandasDataFrameObjectV1)
+def downgrade_pandasdataframeobject_v2_to_v1():
+ return [
+ drop("syft_resolved"),
+ ]
+
+
+@migrate(PandasDataFrameObjectV1, PandasDataFrameObject)
+def upgrade_pandasdataframeobject_v1_to_v2():
+ return [
+ make_set_default("syft_resolved", True),
+ ]
+
+
@serializable()
-class PandasSeriesObject(ActionObject):
+class PandasSeriesObjectV1(ActionObjectV1):
__canonical_name__ = "PandasSeriesObject"
__version__ = SYFT_OBJECT_VERSION_1
syft_internal_type = Series
syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
+
+@serializable()
+class PandasSeriesObject(ActionObject):
+ __canonical_name__ = "PandasSeriesObject"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ syft_internal_type = Series
+ syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
+
# name: Optional[str] = None
# syft_dont_wrap_attrs = ["shape"]
@@ -67,5 +104,19 @@ def syft_is_property(self, obj: Any, method: str) -> bool:
return super().syft_is_property(obj, method)
+@migrate(PandasSeriesObject, PandasSeriesObjectV1)
+def downgrade_pandasseriesframeobject_v2_to_v1():
+ return [
+ drop("syft_resolved"),
+ ]
+
+
+@migrate(PandasSeriesObjectV1, PandasSeriesObject)
+def upgrade_pandasseriesframeobject_v1_to_v2():
+ return [
+ make_set_default("syft_resolved", True),
+ ]
+
+
action_types[DataFrame] = PandasDataFrameObject
action_types[Series] = PandasSeriesObject
diff --git a/packages/syft/src/syft/service/blob_storage/service.py b/packages/syft/src/syft/service/blob_storage/service.py
index 30bf7fbee98..250998c314f 100644
--- a/packages/syft/src/syft/service/blob_storage/service.py
+++ b/packages/syft/src/syft/service/blob_storage/service.py
@@ -4,16 +4,22 @@
from typing import Optional
from typing import Union
+# third party
+import requests
+
# relative
from ...serde.serializable import serializable
+from ...service.action.action_object import ActionObject
from ...store.blob_storage import BlobRetrieval
from ...store.blob_storage.on_disk import OnDiskBlobDeposit
from ...store.blob_storage.seaweedfs import SeaweedFSBlobDeposit
from ...store.document_store import DocumentStore
from ...store.document_store import UIDPartitionKey
+from ...types.blob_storage import BlobFileType
from ...types.blob_storage import BlobStorageEntry
from ...types.blob_storage import BlobStorageMetadata
from ...types.blob_storage import CreateBlobStorageEntry
+from ...types.blob_storage import SecureFilePathLocation
from ...types.uid import UID
from ..context import AuthedServiceContext
from ..response import SyftError
@@ -45,6 +51,82 @@ def get_all_blob_storage_entries(
return result.ok()
return SyftError(message=result.err())
+ @service_method(path="blob_storage.mount_azure", name="mount_azure")
+ def mount_azure(
+ self,
+ context: AuthedServiceContext,
+ account_name: str,
+ account_key: str,
+ container_name: str,
+ bucket_name: str,
+ ):
+ # stdlib
+ import sys
+
+ # TODO: fix arguments
+
+ args_dict = {
+ "account_name": account_name,
+ "account_key": account_key,
+ "container_name": container_name,
+ "remote_name": f"{account_name}{container_name}",
+ "bucket_name": bucket_name,
+ }
+ # TODO: possible wrap this in try catch
+ cfg = context.node.blob_store_config.client_config
+ init_request = requests.post(url=cfg.mount_url, json=args_dict) # nosec
+ print(init_request.content)
+ # TODO check return code
+
+ print(bucket_name, file=sys.stderr)
+
+ res = context.node.blob_storage_client.connect().client.list_objects(
+ Bucket=bucket_name
+ )
+ print(res)
+ objects = res["Contents"]
+ file_sizes = [object["Size"] for object in objects]
+ file_paths = [object["Key"] for object in objects]
+ secure_file_paths = [
+ SecureFilePathLocation(path=file_path) for file_path in file_paths
+ ]
+
+ for sfp, file_size in zip(secure_file_paths, file_sizes):
+ blob_storage_entry = BlobStorageEntry(
+ location=sfp,
+ uploaded_by=context.credentials,
+ file_size=file_size,
+ type_=BlobFileType,
+ bucket_name=bucket_name,
+ )
+ self.stash.set(context.credentials, blob_storage_entry)
+
+ return SyftSuccess(message="Mounting Azure Successful!")
+
+ @service_method(
+ path="blob_storage.get_files_from_bucket", name="get_files_from_bucket"
+ )
+ def get_files_from_bucket(self, context: AuthedServiceContext, bucket_name: str):
+ result = self.stash.find_all(context.credentials, bucket_name=bucket_name)
+ if result.is_err():
+ return result
+ bse_list = result.ok()
+ # stdlib
+ import sys
+
+ print(bse_list, file=sys.stderr)
+ blob_files = []
+ for bse in bse_list:
+ self.stash.set(obj=bse, credentials=context.credentials)
+ blob_file = ActionObject.empty()
+ blob_file.syft_blob_storage_entry_id = bse.id
+ blob_file.syft_client_verify_key = context.credentials
+ blob_file.syft_node_location = context.node.id
+ blob_file.reload_cache()
+ blob_files.append(blob_file.syft_action_data)
+
+ return blob_files
+
@service_method(path="blob_storage.get_by_uid", name="get_by_uid")
def get_blob_storage_entry_by_uid(
self, context: AuthedServiceContext, uid: UID
@@ -79,7 +161,12 @@ def read(
return SyftError(message=f"No blob storage entry exists for uid: {uid}")
with context.node.blob_storage_client.connect() as conn:
- return conn.read(obj.location, obj.type_)
+ res: BlobRetrieval = conn.read(
+ obj.location, obj.type_, bucket_name=obj.bucket_name
+ )
+ res.syft_blob_storage_entry_id = uid
+ res.file_size = obj.file_size
+ return res
return SyftError(message=result.err())
@service_method(
@@ -147,6 +234,7 @@ def mark_write_complete(
context: AuthedServiceContext,
uid: UID,
etags: List,
+ no_lines: Optional[int] = 0,
) -> Union[SyftError, SyftSuccess]:
result = self.stash.get_by_uid(
credentials=context.credentials,
@@ -160,6 +248,14 @@ def mark_write_complete(
if obj is None:
return SyftError(message=f"No blob storage entry exists for uid: {uid}")
+ obj.no_lines = no_lines
+ result = self.stash.update(
+ credentials=context.credentials,
+ obj=obj,
+ )
+ if result.is_err():
+ return SyftError(message=f"{result.err()}")
+
with context.node.blob_storage_client.connect() as conn:
result = conn.complete_multipart_upload(obj, etags)
diff --git a/packages/syft/src/syft/service/code/code_parse.py b/packages/syft/src/syft/service/code/code_parse.py
index 5174cbba261..6e985e35010 100644
--- a/packages/syft/src/syft/service/code/code_parse.py
+++ b/packages/syft/src/syft/service/code/code_parse.py
@@ -1,5 +1,7 @@
# stdlib
+from _ast import Module
import ast
+from typing import Any
class GlobalsVisitor(ast.NodeVisitor):
@@ -7,3 +9,17 @@ def generic_visit(self, node):
if isinstance(node, ast.Global):
raise Exception("No Globals allowed!")
ast.NodeVisitor.generic_visit(self, node)
+
+
+class LaunchJobVisitor(ast.NodeVisitor):
+ def visit_Module(self, node: Module) -> Any:
+ self.nested_calls = []
+ self.generic_visit(node)
+
+ def visit_Call(self, node):
+ if isinstance(node.func, ast.Attribute):
+ if (
+ getattr(node.func.value, "id", None) == "domain"
+ and node.func.attr == "launch_job"
+ ):
+ self.nested_calls.append(node.args[0].id)
diff --git a/packages/syft/src/syft/service/code/user_code.py b/packages/syft/src/syft/service/code/user_code.py
index 092450a43ee..8e6f0ae2d1b 100644
--- a/packages/syft/src/syft/service/code/user_code.py
+++ b/packages/syft/src/syft/service/code/user_code.py
@@ -3,6 +3,7 @@
# stdlib
import ast
+import datetime
from enum import Enum
import hashlib
import inspect
@@ -10,6 +11,7 @@
import itertools
import sys
import time
+import traceback
from typing import Any
from typing import Callable
from typing import Dict
@@ -18,37 +20,51 @@
from typing import Tuple
from typing import Type
from typing import Union
+from typing import final
# third party
from IPython.display import display
+from result import Err
from typing_extensions import Self
# relative
from ...abstract_node import NodeType
+from ...client.api import APIRegistry
from ...client.api import NodeIdentity
+from ...client.client import PythonConnection
from ...client.enclave_client import EnclaveMetadata
from ...node.credentials import SyftVerifyKey
+from ...protocol.data_protocol import get_data_protocol
from ...serde.deserialize import _deserialize
from ...serde.serializable import serializable
from ...serde.serialize import _serialize
from ...store.document_store import PartitionKey
+from ...store.linked_obj import LinkedObject
from ...types.datetime import DateTime
+from ...types.syft_migration import migrate
from ...types.syft_object import SYFT_OBJECT_VERSION_1
+from ...types.syft_object import SYFT_OBJECT_VERSION_2
from ...types.syft_object import SyftHashableObject
from ...types.syft_object import SyftObject
from ...types.transforms import TransformContext
from ...types.transforms import add_node_uid_for_key
+from ...types.transforms import drop
from ...types.transforms import generate_id
+from ...types.transforms import make_set_default
from ...types.transforms import transform
from ...types.uid import UID
from ...util import options
from ...util.colors import SURFACE
from ...util.markdown import CodeMarkdown
from ...util.markdown import as_markdown_code
+from ..action.action_object import Action
+from ..action.action_object import ActionObject
from ..context import AuthedServiceContext
from ..dataset.dataset import Asset
+from ..job.job_stash import Job
from ..policy.policy import CustomInputPolicy
from ..policy.policy import CustomOutputPolicy
+from ..policy.policy import EmpyInputPolicy
from ..policy.policy import ExactMatch
from ..policy.policy import InputPolicy
from ..policy.policy import OutputPolicy
@@ -65,10 +81,13 @@
from ..response import SyftSuccess
from ..response import SyftWarning
from .code_parse import GlobalsVisitor
+from .code_parse import LaunchJobVisitor
from .unparse import unparse
UserVerifyKeyPartitionKey = PartitionKey(key="user_verify_key", type_=SyftVerifyKey)
CodeHashPartitionKey = PartitionKey(key="code_hash", type_=int)
+ServiceFuncNamePartitionKey = PartitionKey(key="service_func_name", type_=str)
+SubmitTimePartitionKey = PartitionKey(key="submit_time", type_=DateTime)
PyCodeObject = Any
@@ -228,7 +247,7 @@ def mutate(
@serializable()
-class UserCode(SyftObject):
+class UserCodeV1(SyftObject):
# version
__canonical_name__ = "UserCode"
__version__ = SYFT_OBJECT_VERSION_1
@@ -254,6 +273,39 @@ class UserCode(SyftObject):
enclave_metadata: Optional[EnclaveMetadata] = None
submit_time: Optional[DateTime]
+
+@serializable()
+class UserCode(SyftObject):
+ # version
+ __canonical_name__ = "UserCode"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ id: UID
+ node_uid: Optional[UID]
+ user_verify_key: SyftVerifyKey
+ raw_code: str
+ input_policy_type: Union[Type[InputPolicy], UserPolicy]
+ input_policy_init_kwargs: Optional[Dict[Any, Any]] = None
+ input_policy_state: bytes = b""
+ output_policy_type: Union[Type[OutputPolicy], UserPolicy]
+ output_policy_init_kwargs: Optional[Dict[Any, Any]] = None
+ output_policy_state: bytes = b""
+ parsed_code: str
+ service_func_name: str
+ unique_func_name: str
+ user_unique_func_name: str
+ code_hash: str
+ signature: inspect.Signature
+ status: UserCodeStatusCollection
+ input_kwargs: List[str]
+ enclave_metadata: Optional[EnclaveMetadata] = None
+ submit_time: Optional[DateTime]
+ uses_domain = (
+ False
+ ) # tracks if the code calls domain.something, variable is set during parsing
+ nested_requests: Dict[str, str] = {}
+ nested_codes: Optional[Dict[str, Tuple[LinkedObject, Dict]]] = {}
+
__attr_searchable__ = ["user_verify_key", "status", "service_func_name"]
__attr_unique__ = []
__repr_attrs__ = ["service_func_name", "input_owners", "code_status"]
@@ -495,7 +547,7 @@ def wrapper(*args: Any, **kwargs: Any) -> Callable:
return wrapper
- def _repr_markdown_(self):
+ def _inner_repr(self, level=0):
shared_with_line = ""
if len(self.output_readers) > 0:
owners_string = " and ".join([f"*{x}*" for x in self.output_reader_names])
@@ -512,8 +564,26 @@ def _repr_markdown_(self):
{shared_with_line}
code:
-{self.raw_code}"""
- return as_markdown_code(md)
+{self.raw_code}
+"""
+ if self.nested_codes != {}:
+ md += """
+
+ Nested Requests:
+ """
+
+ md = "\n".join(
+ [f"{' '*level}{substring}" for substring in md.split("\n")[:-1]]
+ )
+ for _, (obj, _) in self.nested_codes.items():
+ code = obj.resolve
+ md += "\n"
+ md += code._inner_repr(level=level + 1)
+
+ return md
+
+ def _repr_markdown_(self):
+ return as_markdown_code(self._inner_repr())
@property
def show_code(self) -> CodeMarkdown:
@@ -530,11 +600,29 @@ def show_code_cell(self):
ip.set_next_input(warning_message + self.raw_code)
+@migrate(UserCode, UserCodeV1)
+def downgrade_usercode_v2_to_v1():
+ return [
+ drop("uses_domain"),
+ drop("nested_requests"),
+ drop("nested_codes"),
+ ]
+
+
+@migrate(UserCodeV1, UserCode)
+def upgrade_usercode_v1_to_v2():
+ return [
+ make_set_default("uses_domain", False),
+ make_set_default("nested_requests", {}),
+ make_set_default("nested_codes", {}),
+ ]
+
+
@serializable(without=["local_function"])
class SubmitUserCode(SyftObject):
# version
__canonical_name__ = "SubmitUserCode"
- __version__ = SYFT_OBJECT_VERSION_1
+ __version__ = SYFT_OBJECT_VERSION_2
id: Optional[UID]
code: str
@@ -615,10 +703,13 @@ def syft_function_single_use(
def syft_function(
- input_policy: Union[InputPolicy, UID],
+ input_policy: Optional[Union[InputPolicy, UID]] = None,
output_policy: Optional[Union[OutputPolicy, UID]] = None,
share_results_with_owners=False,
) -> SubmitUserCode:
+ if input_policy is None:
+ input_policy = EmpyInputPolicy()
+
if isinstance(input_policy, CustomInputPolicy):
input_policy_type = SubmitUserPolicy.from_obj(input_policy)
else:
@@ -676,10 +767,12 @@ def generate_unique_func_name(context: TransformContext) -> TransformContext:
def process_code(
+ context,
raw_code: str,
func_name: str,
original_func_name: str,
- input_kwargs: List[str],
+ policy_input_kwargs: List[str],
+ function_input_kwargs: List[str],
) -> str:
tree = ast.parse(raw_code)
@@ -690,11 +783,14 @@ def process_code(
f = tree.body[0]
f.decorator_list = []
- keywords = [ast.keyword(arg=i, value=[ast.Name(id=i)]) for i in input_kwargs]
+ call_args = function_input_kwargs
+ if "domain" in function_input_kwargs:
+ context.output["uses_domain"] = True
+ call_stmt_keywords = [ast.keyword(arg=i, value=[ast.Name(id=i)]) for i in call_args]
call_stmt = ast.Assign(
targets=[ast.Name(id="result")],
value=ast.Call(
- func=ast.Name(id=original_func_name), args=[], keywords=keywords
+ func=ast.Name(id=original_func_name), args=[], keywords=call_stmt_keywords
),
lineno=0,
)
@@ -730,16 +826,35 @@ def new_check_code(context: TransformContext) -> TransformContext:
input_keys += d.keys()
processed_code = process_code(
+ context,
raw_code=context.output["raw_code"],
func_name=context.output["unique_func_name"],
original_func_name=context.output["service_func_name"],
- input_kwargs=input_keys,
+ policy_input_kwargs=input_keys,
+ function_input_kwargs=context.output["input_kwargs"],
)
context.output["parsed_code"] = processed_code
return context
+def locate_launch_jobs(context: TransformContext) -> TransformContext:
+ # stdlib
+ nested_requests = {}
+ tree = ast.parse(context.output["raw_code"])
+
+ # look for domain arg
+ if "domain" in [arg.arg for arg in tree.body[0].args.args]:
+ v = LaunchJobVisitor()
+ v.visit(tree)
+ nested_calls = v.nested_calls
+ for call in nested_calls:
+ nested_requests[call] = "latest"
+
+ context.output["nested_requests"] = nested_requests
+ return context
+
+
def compile_byte_code(parsed_code: str) -> Optional[PyCodeObject]:
try:
return compile(parsed_code, " Request time: {self.request_time} Changes: {str_changes} Status: {self.status} Requested on: {node_name} of type \
{metadata.node_type.value.capitalize()} Requested by: {self.requesting_user_name} {email_str} {institution_str} Changes: {str_changes}
"
+ msg += self.nested_repr(node=new_node, level=level + 1)
+ return msg
+
def __repr_syft_nested__(self):
- return f"Request to change {self.link.service_func_name} to permission RequestStatus.APPROVED"
+ msg = f"Request to change {self.link.service_func_name} to permission RequestStatus.APPROVED"
+ if self.nested_solved:
+ if self.link.nested_codes == {}:
+ msg += ". No nested requests"
+ else:
+ msg += ".
This change requests the following nested functions calls:
"
+ msg += self.nested_repr()
+ else:
+ msg += ". Nested Requests not resolved"
+ return msg
def _repr_markdown_(self) -> str:
link = self.link
@@ -805,6 +873,20 @@ def valid(self) -> Union[SyftSuccess, SyftError]:
)
return SyftSuccess(message=f"{type(self)} valid")
+ # def get_nested_requests(self, context, code_tree: Dict[str: Tuple[LinkedObject, Dict]]):
+ # approved_nested_codes = {}
+ # for key, (linked_obj, new_code_tree) in code_tree.items():
+ # code_obj = linked_obj.resolve_with_context(context).ok()
+ # approved_nested_codes[key] = code_obj.id
+
+ # res = self.get_nested_requests(context, new_code_tree)
+ # if isinstance(res, SyftError):
+ # return res
+ # code_obj.nested_codes = res
+ # linked_obj.update_with_context(context, code_obj)
+
+ # return approved_nested_codes
+
def mutate(self, obj: UserCode, context: ChangeContext, undo: bool) -> Any:
reason: str = context.extra_kwargs.get("reason", "")
if not undo:
@@ -814,6 +896,8 @@ def mutate(self, obj: UserCode, context: ChangeContext, undo: bool) -> Any:
node_id=context.node.id,
verify_key=context.node.signing_key.verify_key,
)
+ if isinstance(res, SyftError):
+ return res
else:
res = obj.status.mutate(
value=(UserCodeStatus.DENIED, reason),
@@ -853,6 +937,7 @@ def _run(
from ..enclave.enclave_service import propagate_inputs_to_enclave
user_code = res
+
if self.is_enclave_request(user_code):
enclave_res = propagate_inputs_to_enclave(
user_code=res, context=context
@@ -883,3 +968,17 @@ def link(self) -> Optional[SyftObject]:
if self.linked_obj:
return self.linked_obj.resolve
return None
+
+
+@migrate(UserCodeStatusChange, UserCodeStatusChangeV1)
+def downgrade_usercodestatuschange_v2_to_v1():
+ return [
+ drop("nested_solved"),
+ ]
+
+
+@migrate(UserCodeStatusChangeV1, UserCodeStatusChange)
+def upgrade_usercodestatuschange_v1_to_v2():
+ return [
+ make_set_default("nested_solved", True),
+ ]
diff --git a/packages/syft/src/syft/service/request/request_service.py b/packages/syft/src/syft/service/request/request_service.py
index ac71a7a2726..c0bb7aea4cf 100644
--- a/packages/syft/src/syft/service/request/request_service.py
+++ b/packages/syft/src/syft/service/request/request_service.py
@@ -1,6 +1,8 @@
# stdlib
+from typing import Dict
from typing import List
from typing import Optional
+from typing import Tuple
from typing import Union
# third party
@@ -15,6 +17,7 @@
from ...util.telemetry import instrument
from ..action.action_permissions import ActionObjectPermission
from ..action.action_permissions import ActionPermission
+from ..code.user_code import UserCode
from ..context import AuthedServiceContext
from ..notification.notification_service import CreateNotification
from ..notification.notification_service import NotificationService
@@ -34,6 +37,7 @@
from .request import RequestInfoFilter
from .request import RequestStatus
from .request import SubmitRequest
+from .request import UserCodeStatusChange
from .request_stash import RequestStash
@@ -100,13 +104,56 @@ def submit(
print("Failed to submit Request", e)
raise e
+ def expand_node(self, context: AuthedServiceContext, code_obj: UserCode):
+ user_code_service = context.node.get_service("usercodeservice")
+ nested_requests = user_code_service.solve_nested_requests(context, code_obj)
+
+ new_nested_requests = {}
+ for func_name, code in nested_requests.items():
+ nested_dict = self.expand_node(context, code)
+ if isinstance(nested_dict, SyftError):
+ return nested_dict
+ code.nested_codes = nested_dict
+ res = user_code_service.stash.update(context.credentials, code)
+ if isinstance(res, Err):
+ return res
+ linked_obj = LinkedObject.from_obj(code, node_uid=context.node.id)
+ new_nested_requests[func_name] = (linked_obj, nested_dict)
+
+ return new_nested_requests
+
+ def resolve_nested_requests(self, context, request):
+ # TODO: change this if we have more UserCode Changes
+ if len(request.changes) != 1:
+ return request
+
+ change = request.changes[0]
+ if isinstance(change, UserCodeStatusChange):
+ if change.nested_solved:
+ return request
+ code_obj = change.linked_obj.resolve_with_context(context=context).ok()
+ # recursively check what other UserCodes to approve
+ nested_requests: Dict[str : Tuple[LinkedObject, Dict]] = self.expand_node(
+ context, code_obj
+ )
+ if isinstance(nested_requests, Err):
+ return SyftError(message=nested_requests.value)
+ change.nested_solved = True
+ code_obj.nested_codes = nested_requests
+ change.linked_obj.update_with_context(context=context, obj=code_obj)
+
+ request.changes = [change]
+ new_request = self.save(context=context, request=request)
+ return new_request
+ return request
+
@service_method(path="request.get_all", name="get_all")
def get_all(self, context: AuthedServiceContext) -> Union[List[Request], SyftError]:
result = self.stash.get_all(context.credentials)
if result.is_err():
return SyftError(message=str(result.err()))
requests = result.ok()
- return requests
+ return [self.resolve_nested_requests(context, request) for request in requests]
@service_method(path="request.get_all_info", name="get_all_info")
def get_all_info(
diff --git a/packages/syft/src/syft/service/response.py b/packages/syft/src/syft/service/response.py
index fee752d6165..5b6c88ebc74 100644
--- a/packages/syft/src/syft/service/response.py
+++ b/packages/syft/src/syft/service/response.py
@@ -3,6 +3,9 @@
import traceback
from typing import Any
+# third party
+from result import Err
+
# relative
from ..serde.serializable import serializable
from ..types.base import SyftBaseModel
@@ -46,6 +49,9 @@ class SyftError(SyftResponseMessage):
def _repr_html_class_(self) -> str:
return "alert-danger"
+ def to_result(self):
+ return Err(value=self.message)
+
@serializable()
class SyftSuccess(SyftResponseMessage):
diff --git a/packages/syft/src/syft/service/service.py b/packages/syft/src/syft/service/service.py
index 52fb081b541..edd5482b9aa 100644
--- a/packages/syft/src/syft/service/service.py
+++ b/packages/syft/src/syft/service/service.py
@@ -478,3 +478,5 @@ def from_api_or_context(
_private_api_path,
)
return partial(service_method, node_context)
+ else:
+ print("Could not get method from api or context")
diff --git a/packages/syft/src/syft/service/worker/__init__.py b/packages/syft/src/syft/service/worker/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/packages/syft/src/syft/service/worker/worker_service.py b/packages/syft/src/syft/service/worker/worker_service.py
new file mode 100644
index 00000000000..2ba77ff8e98
--- /dev/null
+++ b/packages/syft/src/syft/service/worker/worker_service.py
@@ -0,0 +1,258 @@
+# stdlib
+import socket
+from typing import List
+from typing import Union
+
+# third party
+import docker
+
+# relative
+from ...serde.serializable import serializable
+from ...store.document_store import BaseUIDStoreStash
+from ...store.document_store import DocumentStore
+from ...store.document_store import PartitionSettings
+from ...store.document_store import SYFT_OBJECT_VERSION_1
+from ...store.document_store import SyftObject
+from ...store.document_store import SyftSuccess
+from ...types.datetime import DateTime
+from ...util.telemetry import instrument
+from ..service import AbstractService
+from ..service import AuthedServiceContext
+from ..service import SyftError
+from ..service import service_method
+from ..user.user_roles import ADMIN_ROLE_LEVEL
+
+
+@serializable()
+class DockerWorker(SyftObject):
+ # version
+ __canonical_name__ = "ContainerImage"
+ __version__ = SYFT_OBJECT_VERSION_1
+
+ __attr_searchable__ = ["container_id"]
+ __attr_unique__ = ["container_id"]
+ __repr_attrs__ = ["container_id", "created_at"]
+
+ container_id: str
+ created_at: DateTime
+
+
+@instrument
+@serializable()
+class WorkerStash(BaseUIDStoreStash):
+ object_type = DockerWorker
+ settings: PartitionSettings = PartitionSettings(
+ name=DockerWorker.__canonical_name__, object_type=DockerWorker
+ )
+
+ def __init__(self, store: DocumentStore) -> None:
+ super().__init__(store=store)
+
+
+# def get_default_env_vars(context: AuthedServiceContext):
+# if context.node.runs_in_docker:
+# # get env vars from current environment
+# return dict(os.environ)
+# else:
+# # read env vars from .env file
+# env_path = f"{context.node.host_syft_location}/packages/grid/.env"
+# with open(env_path) as f:
+# lines = f.read().splitlines()
+
+# default_env_vars = {}
+# for line in lines:
+# if "=" in line:
+# try:
+# var_name, value = line.split("=", 1)
+
+# def remove_redundant_quotes(value):
+# for s in ['"', "'"]:
+# if len(value) != 0:
+# if value[0] == s:
+# value = value[1:]
+# if value[-1] == s:
+# value = value[:-1]
+
+# value = remove_redundant_quotes(value)
+# default_env_vars[var_name] = value
+# except Exception as e:
+# print("error parsing env file", e)
+# return default_env_vars
+
+
+# PORT_COUNTER = 0
+
+
+# def get_env_vars(context: AuthedServiceContext):
+# default_env_vars = get_default_env_vars(context)
+# # stdlib
+# import secrets
+
+# worker_tag = "".join([str(secrets.choice(list(range(10)))) for i in range(10)])
+# node = context.node
+# # TODO, improve
+# global PORT_COUNTER
+# PORT_COUNTER += 1
+# extra_env_vars = {
+# "SERVICE_NAME": "backend",
+# "CREATE_PRODUCER": "false",
+# "N_CONSUMERS": "1",
+# "DEV_MODE": node.dev_mode,
+# "DEFAULT_ROOT_USERNAME": f"worker-{worker_tag}",
+# "PORT": str(8003 + PORT_COUNTER),
+# "QUEUE_PORT": node.queue_config.client_config.queue_port,
+# "HTTP_PORT": str(88 + PORT_COUNTER),
+# "HTTPS_PORT": str(446 + PORT_COUNTER),
+# "DEFAULT_ROOT_EMAIL": f"{worker_tag}@openmined.org",
+# }
+# # if node.dev_mode:
+# # extra_env_vars["WATCHFILES_FORCE_POLLING"] = "true"
+
+# result = {**default_env_vars, **extra_env_vars}
+# result.pop("NODE_PRIVATE_KEY", None)
+# return result
+
+
+def get_main_backend() -> str:
+ hostname = socket.gethostname()
+ return f"{hostname}-backend-1"
+
+
+def start_worker_container(worker_num: int, context: AuthedServiceContext):
+ client = docker.from_env()
+ existing_container_name = get_main_backend()
+ hostname = socket.gethostname()
+ worker_name = f"{hostname}-worker-{worker_num}"
+ return create_new_container_from_existing(
+ worker_name=worker_name,
+ client=client,
+ existing_container_name=existing_container_name,
+ )
+
+
+def create_new_container_from_existing(
+ worker_name: str, client: docker.client.DockerClient, existing_container_name: str
+) -> docker.models.containers.Container:
+ # Get the existing container
+ existing_container = client.containers.get(existing_container_name)
+
+ # Inspect the existing container
+ details = existing_container.attrs
+
+ # Extract relevant settings
+ image = details["Config"]["Image"]
+ command = details["Config"]["Cmd"]
+ environment = details["Config"]["Env"]
+ ports = details["NetworkSettings"]["Ports"]
+ host_config = details["HostConfig"]
+
+ volumes = {}
+ for vol in host_config["Binds"]:
+ parts = vol.split(":")
+ key = parts[0]
+ bind = parts[1]
+ mode = parts[2]
+ if "/storage" in bind:
+ # we need this because otherwise we are using the same node private key
+ # which will make account creation fail
+ worker_postfix = worker_name.split("-", 1)[1]
+ key = f"{key}-{worker_postfix}"
+ volumes[key] = {"bind": bind, "mode": mode}
+
+ # we need this because otherwise we are using the same node private key
+ # which will make account creation fail
+
+ environment = dict([e.split("=", 1) for e in environment])
+ environment["CREATE_PRODUCER"] = "false"
+ environment["N_CONSUMERS"] = 1
+ environment["DEFAULT_ROOT_USERNAME"] = worker_name
+ environment["DEFAULT_ROOT_EMAIL"] = f"{worker_name}@openmined.org"
+ environment["PORT"] = str(8003 + WORKER_NUM)
+ environment["HTTP_PORT"] = str(88 + WORKER_NUM)
+ environment["HTTPS_PORT"] = str(446 + WORKER_NUM)
+ environment.pop("NODE_PRIVATE_KEY", None)
+
+ new_container = client.containers.create(
+ name=worker_name,
+ image=image,
+ command=command,
+ environment=environment,
+ ports=ports,
+ detach=True,
+ volumes=volumes,
+ tty=True,
+ stdin_open=True,
+ network_mode=f"container:{existing_container.id}",
+ )
+
+ new_container.start()
+ return new_container
+
+
+WORKER_NUM = 0
+
+
+@instrument
+@serializable()
+class WorkerService(AbstractService):
+ store: DocumentStore
+ stash: WorkerStash
+
+ def __init__(self, store: DocumentStore) -> None:
+ self.store = store
+ self.stash = WorkerStash(store=store)
+
+ @service_method(
+ path="worker.start_workers", name="start_workers", roles=ADMIN_ROLE_LEVEL
+ )
+ def start_workers(
+ self, context: AuthedServiceContext, n: int = 1
+ ) -> Union[SyftSuccess, SyftError]:
+ """Add a Container Image."""
+ for _worker_num in range(n):
+ global WORKER_NUM
+ WORKER_NUM += 1
+ res = start_worker_container(WORKER_NUM, context)
+ obj = DockerWorker(container_id=res.id, created_at=DateTime.now())
+ result = self.stash.set(context.credentials, obj)
+ if result.is_err():
+ return SyftError(message=f"Failed to start worker. {result.err()}")
+
+ return SyftSuccess(message=f"{n} workers added")
+
+ @service_method(path="worker.list", name="list", roles=ADMIN_ROLE_LEVEL)
+ def list(self, context: AuthedServiceContext) -> Union[SyftSuccess, SyftError]:
+ """Add a Container Image."""
+ result = self.stash.get_all(context.credentials)
+
+ if result.is_err():
+ return SyftError(message=f"Failed to fetch workers. {result.err()}")
+ else:
+ return result.ok()
+
+ @service_method(path="worker.stop", name="stop", roles=ADMIN_ROLE_LEVEL)
+ def stop(
+ self,
+ context: AuthedServiceContext,
+ workers: Union[List[DockerWorker], DockerWorker],
+ ) -> Union[SyftSuccess, SyftError]:
+ # listify
+ if isinstance(workers, DockerWorker):
+ workers = [workers]
+
+ client = docker.from_env()
+ for w in workers:
+ result = self.stash.delete_by_uid(context.credentials, uid=w.id)
+
+ if result.is_err():
+ return SyftError(message=f"Failed to stop workers {result.err()}")
+
+ # stop container
+ try:
+ client.containers.list(filters={"id": w.container_id})[0].stop()
+ # also prune here?
+ except Exception as e:
+ # we dont throw an error here because apparently the container was already killed
+ print(f"Failed to kill container {e}")
+
+ return SyftSuccess(message=f"{len(workers)} workers stopped")
diff --git a/packages/syft/src/syft/store/blob_storage/__init__.py b/packages/syft/src/syft/store/blob_storage/__init__.py
index 561fdd4a6b5..48699df9db1 100644
--- a/packages/syft/src/syft/store/blob_storage/__init__.py
+++ b/packages/syft/src/syft/store/blob_storage/__init__.py
@@ -45,7 +45,6 @@
from typing import Optional
from typing import Type
from typing import Union
-from urllib.request import urlretrieve
# third party
from pydantic import BaseModel
@@ -64,47 +63,131 @@
from ...types.blob_storage import CreateBlobStorageEntry
from ...types.blob_storage import SecureFilePathLocation
from ...types.grid_url import GridURL
+from ...types.syft_migration import migrate
from ...types.syft_object import SYFT_OBJECT_VERSION_1
+from ...types.syft_object import SYFT_OBJECT_VERSION_2
from ...types.syft_object import SyftObject
+from ...types.transforms import drop
+from ...types.transforms import make_set_default
from ...types.uid import UID
-from ...util.constants import DEFAULT_TIMEOUT
@serializable()
-class BlobRetrieval(SyftObject):
+class BlobRetrievalV1(SyftObject):
__canonical_name__ = "BlobRetrieval"
__version__ = SYFT_OBJECT_VERSION_1
type_: Optional[Type]
file_name: str
+
+@serializable()
+class BlobRetrieval(SyftObject):
+ __canonical_name__ = "BlobRetrieval"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ type_: Optional[Type]
+ file_name: str
+ syft_blob_storage_entry_id: Optional[UID] = None
+ file_size: Optional[int]
+
def read(self) -> Union[SyftObject, SyftError]:
- pass
+ # we need both methods bcs of inheritrance
+ return self._read()
+
+ def _read(self):
+ with open(self.file_name, "rb") as f:
+ return f.read()
+
+ def _read_data(self, **kwargs):
+ return self._read()
+
+
+@migrate(BlobRetrieval, BlobRetrievalV1)
+def downgrade_blobretrival_v2_to_v1():
+ return [
+ drop(["syft_blob_storage_entry_id", "file_size"]),
+ ]
+
+
+@migrate(BlobRetrievalV1, BlobRetrieval)
+def upgrade_blobretrieval_v1_to_v2():
+ return [
+ make_set_default("syft_blob_storage_entry_id", None),
+ make_set_default("file_size", 1),
+ ]
@serializable()
-class SyftObjectRetrieval(BlobRetrieval):
+class SyftObjectRetrievalV1(BlobRetrievalV1):
__canonical_name__ = "SyftObjectRetrieval"
__version__ = SYFT_OBJECT_VERSION_1
syft_object: bytes
+
+@serializable()
+class SyftObjectRetrieval(BlobRetrieval):
+ __canonical_name__ = "SyftObjectRetrieval"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ syft_object: bytes
+
def read(self) -> Union[SyftObject, SyftError]:
if self.type_ is BlobFileType:
with open(self.file_name, "wb") as fp:
fp.write(self.syft_object)
- return BlobFile(file_name=self.file_name)
+ return BlobFile(
+ file_name=self.file_name,
+ syft_blob_storage_entry_id=self.syft_blob_storage_entry_id,
+ syft_node_location=self.syft_node_location,
+ syft_client_verify_key=self.syft_client_verify_key,
+ )
return deserialize(self.syft_object, from_bytes=True)
+@migrate(SyftObjectRetrieval, SyftObjectRetrievalV1)
+def downgrade_syftobjretrival_v2_to_v1():
+ return [
+ drop(["syft_blob_storage_entry_id", "file_size"]),
+ ]
+
+
+@migrate(SyftObjectRetrievalV1, SyftObjectRetrieval)
+def upgrade_syftobjretrival_v1_to_v2():
+ return [
+ make_set_default("syft_blob_storage_entry_id", None),
+ make_set_default("file_size", 1),
+ ]
+
+
+class BlobRetrievalByURLV1(BlobRetrievalV1):
+ __canonical_name__ = "BlobRetrievalByURL"
+ __version__ = SYFT_OBJECT_VERSION_1
+
+ url: GridURL
+
+
@serializable()
class BlobRetrievalByURL(BlobRetrieval):
__canonical_name__ = "BlobRetrievalByURL"
- __version__ = SYFT_OBJECT_VERSION_1
+ __version__ = SYFT_OBJECT_VERSION_2
url: GridURL
def read(self) -> Union[SyftObject, SyftError]:
+ if self.type_ is BlobFileType:
+ return BlobFile(
+ file_name=self.file_name,
+ syft_client_verify_key=self.syft_client_verify_key,
+ syft_node_location=self.syft_node_location,
+ syft_blob_storage_entry_id=self.syft_blob_storage_entry_id,
+ file_size=self.file_size,
+ )
+ else:
+ return self._read_data()
+
+ def _read_data(self, stream=False, chunk_size=512):
# relative
from ...client.api import APIRegistry
@@ -113,20 +196,39 @@ def read(self) -> Union[SyftObject, SyftError]:
user_verify_key=self.syft_client_verify_key,
)
if api is not None:
- blob_url = api.connection.to_blob_route(self.url.url_path)
+ blob_url = api.connection.to_blob_route(
+ self.url.url_path, host=self.url.host_or_ip
+ )
else:
blob_url = self.url
try:
- if self.type_ is BlobFileType:
- urlretrieve(str(blob_url), filename=self.file_name) # nosec
- return BlobFile(file_name=self.file_name)
- response = requests.get(str(blob_url), timeout=DEFAULT_TIMEOUT)
+ response = requests.get(str(blob_url), stream=stream) # nosec
response.raise_for_status()
+ if self.type_ is BlobFileType:
+ if stream:
+ return response.iter_lines(chunk_size=chunk_size)
+ else:
+ return response.content
return deserialize(response.content, from_bytes=True)
except requests.RequestException as e:
return SyftError(message=f"Failed to retrieve with Error: {e}")
+@migrate(BlobRetrievalByURL, BlobRetrievalByURLV1)
+def downgrade_blobretrivalbyurl_v2_to_v1():
+ return [
+ drop(["syft_blob_storage_entry_id", "file_size"]),
+ ]
+
+
+@migrate(BlobRetrievalByURLV1, BlobRetrievalByURL)
+def upgrade_blobretrivalbyurl_v1_to_v2():
+ return [
+ make_set_default("syft_blob_storage_entry_id", None),
+ make_set_default("file_size", 1),
+ ]
+
+
@serializable()
class BlobDeposit(SyftObject):
__canonical_name__ = "BlobDeposit"
diff --git a/packages/syft/src/syft/store/blob_storage/on_disk.py b/packages/syft/src/syft/store/blob_storage/on_disk.py
index 81f990ca6e8..ca76832ff4c 100644
--- a/packages/syft/src/syft/store/blob_storage/on_disk.py
+++ b/packages/syft/src/syft/store/blob_storage/on_disk.py
@@ -56,7 +56,9 @@ def __enter__(self) -> Self:
def __exit__(self, *exc) -> None:
pass
- def read(self, fp: SecureFilePathLocation, type_: Optional[Type]) -> BlobRetrieval:
+ def read(
+ self, fp: SecureFilePathLocation, type_: Optional[Type], **kwargs
+ ) -> BlobRetrieval:
file_path = self._base_directory / fp.path
return SyftObjectRetrieval(
syft_object=file_path.read_bytes(),
diff --git a/packages/syft/src/syft/store/blob_storage/seaweedfs.py b/packages/syft/src/syft/store/blob_storage/seaweedfs.py
index 8321532920a..c1314663d46 100644
--- a/packages/syft/src/syft/store/blob_storage/seaweedfs.py
+++ b/packages/syft/src/syft/store/blob_storage/seaweedfs.py
@@ -69,12 +69,16 @@ def write(self, data: BytesIO) -> Union[SyftSuccess, SyftError]:
etags = []
try:
+ no_lines = 0
for part_no, (byte_chunk, url) in enumerate(
zip(_byte_chunks(data, DEFAULT_CHUNK_SIZE), self.urls),
start=1,
):
+ no_lines += byte_chunk.count(b"\n")
if api is not None:
- blob_url = api.connection.to_blob_route(url.url_path)
+ blob_url = api.connection.to_blob_route(
+ url.url_path, host=url.host_or_ip
+ )
else:
blob_url = url
response = requests.put(
@@ -92,8 +96,7 @@ def write(self, data: BytesIO) -> Union[SyftSuccess, SyftError]:
syft_client_verify_key=self.syft_client_verify_key,
)
return mark_write_complete_method(
- etags=etags,
- uid=self.blob_storage_entry_id,
+ etags=etags, uid=self.blob_storage_entry_id, no_lines=no_lines
)
@@ -101,16 +104,23 @@ def write(self, data: BytesIO) -> Union[SyftSuccess, SyftError]:
class SeaweedFSClientConfig(BlobStorageClientConfig):
host: str
port: int
+ mount_port: Optional[int] = None
access_key: str
secret_key: str
region: str
- bucket_name: str
+ default_bucket_name: str = "defaultbucket"
@property
def endpoint_url(self) -> str:
grid_url = GridURL(host_or_ip=self.host, port=self.port)
return grid_url.url
+ @property
+ def mount_url(self) -> str:
+ if self.mount_port is None:
+ raise ValueError("Seaweed should be configured with a mount port to mount")
+ return f"http://{self.host}:{self.mount_port}/configure_azure"
+
@serializable()
class SeaweedFSClient(BlobStorageClient):
@@ -126,18 +136,18 @@ def connect(self) -> BlobStorageConnection:
config=Config(signature_version="s3v4"),
region_name=self.config.region,
),
- bucket_name=self.config.bucket_name,
+ default_bucket_name=self.config.default_bucket_name,
)
@serializable()
class SeaweedFSConnection(BlobStorageConnection):
client: S3BaseClient
- bucket_name: str
+ default_bucket_name: str
- def __init__(self, client: S3BaseClient, bucket_name: str):
+ def __init__(self, client: S3BaseClient, default_bucket_name: str):
self.client = client
- self.bucket_name = bucket_name
+ self.default_bucket_name = default_bucket_name
def __enter__(self) -> Self:
return self
@@ -145,11 +155,15 @@ def __enter__(self) -> Self:
def __exit__(self, *exc) -> None:
self.client.close()
- def read(self, fp: SecureFilePathLocation, type_: Optional[Type]) -> BlobRetrieval:
+ def read(
+ self, fp: SecureFilePathLocation, type_: Optional[Type], bucket_name=None
+ ) -> BlobRetrieval:
+ if bucket_name is None:
+ bucket_name = self.default_bucket_name
try:
url = self.client.generate_presigned_url(
ClientMethod="get_object",
- Params={"Bucket": self.bucket_name, "Key": fp.path},
+ Params={"Bucket": bucket_name, "Key": fp.path},
ExpiresIn=READ_EXPIRATION_TIME,
)
@@ -165,7 +179,7 @@ def allocate(
try:
file_name = obj.file_name
result = self.client.create_multipart_upload(
- Bucket=self.bucket_name,
+ Bucket=self.default_bucket_name,
Key=file_name,
)
upload_id = result["UploadId"]
@@ -183,7 +197,7 @@ def write(self, obj: BlobStorageEntry) -> BlobDeposit:
self.client.generate_presigned_url(
ClientMethod="upload_part",
Params={
- "Bucket": self.bucket_name,
+ "Bucket": self.default_bucket_name,
"Key": obj.location.path,
"UploadId": obj.location.upload_id,
"PartNumber": i + 1,
@@ -203,7 +217,7 @@ def complete_multipart_upload(
) -> Union[SyftError, SyftSuccess]:
try:
self.client.complete_multipart_upload(
- Bucket=self.bucket_name,
+ Bucket=self.default_bucket_name,
Key=blob_entry.location.path,
MultipartUpload={"Parts": etags},
UploadId=blob_entry.location.upload_id,
@@ -217,7 +231,7 @@ def delete(
fp: SecureFilePathLocation,
) -> Union[SyftSuccess, SyftError]:
try:
- self.client.delete_object(Bucket=self.bucket_name, Key=fp.path)
+ self.client.delete_object(Bucket=self.default_bucket_name, Key=fp.path)
return SyftSuccess(message="Successfully deleted file.")
except BotoClientError as e:
return SyftError(message=str(e))
diff --git a/packages/syft/src/syft/store/document_store.py b/packages/syft/src/syft/store/document_store.py
index 474975a5511..efad2a9a7d1 100644
--- a/packages/syft/src/syft/store/document_store.py
+++ b/packages/syft/src/syft/store/document_store.py
@@ -355,6 +355,7 @@ def store_query_keys(self, objs: Any) -> QueryKeys:
def _thread_safe_cbk(self, cbk: Callable, *args, **kwargs):
locked = self.lock.acquire(blocking=True)
if not locked:
+ print("FAILED TO LOCK")
return Err("Failed to acquire lock for the operation")
try:
diff --git a/packages/syft/src/syft/store/linked_obj.py b/packages/syft/src/syft/store/linked_obj.py
index 8ac219a3936..97611d56b64 100644
--- a/packages/syft/src/syft/store/linked_obj.py
+++ b/packages/syft/src/syft/store/linked_obj.py
@@ -63,8 +63,7 @@ def update_with_context(
result = context.node.get_service(self.service_type).stash.update(
credentials, obj
)
- if result.is_ok():
- return result
+ return result
@classmethod
def from_obj(
diff --git a/packages/syft/src/syft/store/locks.py b/packages/syft/src/syft/store/locks.py
index 0714e3b0026..a32bcd67c8d 100644
--- a/packages/syft/src/syft/store/locks.py
+++ b/packages/syft/src/syft/store/locks.py
@@ -1,10 +1,12 @@
# stdlib
+from collections import defaultdict
import datetime
import json
from pathlib import Path
import threading
import time
from typing import Callable
+from typing import Dict
from typing import Optional
import uuid
@@ -17,7 +19,8 @@
# relative
from ..serde.serializable import serializable
-from ..util.logger import debug
+
+THREAD_FILE_LOCKS: Dict[int, Dict[str, int]] = defaultdict(dict)
@serializable()
@@ -190,7 +193,8 @@ def _thread_safe_cbk(self, cbk: Callable) -> bool:
try:
result = cbk()
- except BaseException:
+ except BaseException as e:
+ print(e)
result = False
self._lock_py_thread._release()
@@ -200,7 +204,8 @@ def _acquire(self) -> bool:
return self._thread_safe_cbk(self._acquire_file_lock)
def _release(self) -> None:
- return self._thread_safe_cbk(self._release_file_lock)
+ res = self._thread_safe_cbk(self._release_file_lock)
+ return res
def _acquire_file_lock(self) -> bool:
if not self._lock_file_enabled:
@@ -217,6 +222,8 @@ def _acquire_file_lock(self) -> bool:
break
except BaseException:
time.sleep(0.1)
+ if _retry == 9:
+ pass
now = self._now()
has_expired = self._has_expired(data, now)
@@ -382,11 +389,10 @@ def acquire(self, blocking: bool = True) -> bool:
elapsed = time.time() - start_time
else:
return True
- debug(
- "Timeout elapsed after %s seconds "
- "while trying to acquiring "
- "lock." % self.timeout
+ print(
+ f"Timeout elapsed after {self.timeout} seconds while trying to acquiring lock."
)
+ # third party
return False
def _acquire(self) -> bool:
diff --git a/packages/syft/src/syft/store/mongo_document_store.py b/packages/syft/src/syft/store/mongo_document_store.py
index c502db794d2..efdd6496154 100644
--- a/packages/syft/src/syft/store/mongo_document_store.py
+++ b/packages/syft/src/syft/store/mongo_document_store.py
@@ -231,6 +231,9 @@ def permissions(self) -> Result[MongoCollection, Err]:
return Ok(self._permissions)
+ def set(self, *args, **kwargs):
+ return self._set(*args, **kwargs)
+
def _set(
self,
credentials: SyftVerifyKey,
@@ -357,6 +360,11 @@ def _find_index_or_search_keys(
credentials=credentials, qks=qks, order_by=order_by
)
+ @property
+ def data(self):
+ values: List = self._all(credentials=None, has_permission=True).ok()
+ return {v.id: v for v in values}
+
def _get_all_from_store(
self,
credentials: SyftVerifyKey,
@@ -383,7 +391,9 @@ def _get_all_from_store(
# TODO: maybe do this in loop before this
res = []
for s in syft_objs:
- if self.has_permission(ActionObjectREAD(uid=s.id, credentials=credentials)):
+ if has_permission or self.has_permission(
+ ActionObjectREAD(uid=s.id, credentials=credentials)
+ ):
res.append(s)
return Ok(res)
diff --git a/packages/syft/src/syft/store/sqlite_document_store.py b/packages/syft/src/syft/store/sqlite_document_store.py
index 8a7600ae36d..2ec313a1a8a 100644
--- a/packages/syft/src/syft/store/sqlite_document_store.py
+++ b/packages/syft/src/syft/store/sqlite_document_store.py
@@ -2,6 +2,7 @@
from __future__ import annotations
# stdlib
+from collections import defaultdict
from copy import deepcopy
from pathlib import Path
import sqlite3
@@ -35,6 +36,20 @@
from .kv_document_store import KeyValueStorePartition
from .locks import FileLockingConfig
from .locks import LockingConfig
+from .locks import SyftLock
+
+# here we can create a single connection per cache_key
+# since pytest is concurrent processes, we need to isolate each connection
+# by its filename and optionally the thread that its running in
+# we keep track of each SQLiteBackingStore init in REF_COUNTS
+# when it hits 0 we can close the connection and release the file descriptor
+SQLITE_CONNECTION_POOL_DB: Dict[str, sqlite3.Connection] = {}
+SQLITE_CONNECTION_POOL_CUR: Dict[str, sqlite3.Cursor] = {}
+REF_COUNTS: Dict[str, int] = defaultdict(int)
+
+
+def cache_key(db_name: str) -> str:
+ return f"{db_name}_{thread_ident()}"
def _repr_debug_(value: Any) -> str:
@@ -43,6 +58,23 @@ def _repr_debug_(value: Any) -> str:
return repr(value)
+def raise_exception(table_name: str, e: Exception):
+ if "disk I/O error" in str(e):
+ message = f"Error usually related to concurrent writes. {str(e)}"
+ raise Exception(message)
+
+ if "Cannot operate on a closed database" in str(e):
+ message = (
+ "Error usually related to calling self.db.close()"
+ + f"before last SQLiteBackingStore.__del__ gets called. {str(e)}"
+ )
+ raise Exception(message)
+
+ # if its something else other than "table already exists" raise original e
+ if f"table {table_name} already exists" not in str(e):
+ raise e
+
+
@serializable(attrs=["index_name", "settings", "store_config"])
class SQLiteBackingStore(KeyValueBackingStore):
"""Core Store logic for the SQLite stores.
@@ -69,9 +101,16 @@ def __init__(
self.settings = settings
self.store_config = store_config
self._ddtype = ddtype
- self._db: Dict[int, sqlite3.Connection] = {}
- self._cur: Dict[int, sqlite3.Cursor] = {}
+ self.file_path = self.store_config.client_config.file_path
+ self.db_filename = store_config.client_config.filename
+
+ # if tempfile.TemporaryDirectory() varies from process to process
+ # could this cause different locks on the same file
+ temp_dir = tempfile.TemporaryDirectory().name
+ lock_path = Path(temp_dir) / "sqlite_locks" / self.db_filename
+ self.lock_config = FileLockingConfig(client_path=lock_path)
self.create_table()
+ REF_COUNTS[cache_key(self.db_filename)] += 1
@property
def table_name(self) -> str:
@@ -83,50 +122,58 @@ def _connect(self) -> None:
# there will be many threads handling incoming requests so we need to ensure
# that different connections are used in each thread. By using a dict for the
# _db and _cur we can ensure they are never shared
- self.file_path = self.store_config.client_config.file_path
path = Path(self.file_path)
if not path.exists():
path.parent.mkdir(parents=True, exist_ok=True)
- self._db[thread_ident()] = sqlite3.connect(
+ connection = sqlite3.connect(
self.file_path,
timeout=self.store_config.client_config.timeout,
- check_same_thread=self.store_config.client_config.check_same_thread,
+ check_same_thread=False, # do we need this if we use the lock?
+ # check_same_thread=self.store_config.client_config.check_same_thread,
)
-
# TODO: Review OSX compatibility.
# Set journal mode to WAL.
- # self._db[thread_ident()].execute("pragma journal_mode=wal")
+ # connection.execute("pragma journal_mode=wal")
+ SQLITE_CONNECTION_POOL_DB[cache_key(self.db_filename)] = connection
def create_table(self) -> None:
try:
- self.cur.execute(
- f"create table {self.table_name} (uid VARCHAR(32) NOT NULL PRIMARY KEY, " # nosec
- + "repr TEXT NOT NULL, value BLOB NOT NULL, " # nosec
- + "sqltime TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL)" # nosec
- )
- self.db.commit()
- except sqlite3.OperationalError as e:
- if f"table {self.table_name} already exists" not in str(e):
- raise e
+ with SyftLock(self.lock_config):
+ self.cur.execute(
+ f"create table {self.table_name} (uid VARCHAR(32) NOT NULL PRIMARY KEY, " # nosec
+ + "repr TEXT NOT NULL, value BLOB NOT NULL, " # nosec
+ + "sqltime TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL)" # nosec
+ )
+ self.db.commit()
+ except Exception as e:
+ raise_exception(self.table_name, e)
@property
def db(self) -> sqlite3.Connection:
- if thread_ident() not in self._db:
+ if cache_key(self.db_filename) not in SQLITE_CONNECTION_POOL_DB:
self._connect()
- return self._db[thread_ident()]
+ return SQLITE_CONNECTION_POOL_DB[cache_key(self.db_filename)]
@property
def cur(self) -> sqlite3.Cursor:
- if thread_ident() not in self._cur:
- self._cur[thread_ident()] = self.db.cursor()
+ if cache_key(self.db_filename) not in SQLITE_CONNECTION_POOL_CUR:
+ SQLITE_CONNECTION_POOL_CUR[cache_key(self.db_filename)] = self.db.cursor()
- return self._cur[thread_ident()]
+ return SQLITE_CONNECTION_POOL_CUR[cache_key(self.db_filename)]
def _close(self) -> None:
self._commit()
- self.db.close()
+ REF_COUNTS[cache_key(self.db_filename)] -= 1
+ if REF_COUNTS[cache_key(self.db_filename)] <= 0:
+ # once you close it seems like other object references can't re-use the
+ # same connection
+ self.db.close()
+ del SQLITE_CONNECTION_POOL_DB[cache_key(self.db_filename)]
+ else:
+ # don't close yet because another SQLiteBackingStore is probably still open
+ pass
def _commit(self) -> None:
self.db.commit()
@@ -134,20 +181,25 @@ def _commit(self) -> None:
def _execute(
self, sql: str, *args: Optional[List[Any]]
) -> Result[Ok[sqlite3.Cursor], Err[str]]:
- cursor: Optional[sqlite3.Cursor] = None
- err = None
- try:
- cursor = self.cur.execute(sql, *args)
- except BaseException as e:
- self.db.rollback() # Roll back all changes if an exception occurs.
- err = Err(str(e))
- else:
+ with SyftLock(self.lock_config):
+ cursor: Optional[sqlite3.Cursor] = None
+ err = None
+ try:
+ cursor = self.cur.execute(sql, *args)
+ except Exception as e:
+ raise_exception(self.table_name, e)
+
+ # TODO: Which exception is safe to rollback on?
+ # we should map out some more clear exceptions that can be returned
+ # rather than halting the program like disk I/O error etc
+ # self.db.rollback() # Roll back all changes if an exception occurs.
+ # err = Err(str(e))
self.db.commit() # Commit if everything went ok
- if err is not None:
- return err
+ if err is not None:
+ return err
- return Ok(cursor)
+ return Ok(cursor)
def _set(self, key: UID, value: Any) -> None:
if self._exists(key):
@@ -292,7 +344,6 @@ def items(self) -> Any:
return self._get_all().items()
def pop(self, key: Any) -> Self:
- # NOTE: not thread-safe
value = self._get(key)
self._delete(key)
return value
@@ -307,6 +358,7 @@ def __del__(self):
try:
self._close()
except BaseException:
+ print("Could not close connection")
pass
@@ -324,9 +376,11 @@ class SQLiteStorePartition(KeyValueStorePartition):
def close(self) -> None:
self.lock.acquire()
try:
- self.data._close()
- self.unique_keys._close()
- self.searchable_keys._close()
+ # I think we don't want these now, because of the REF_COUNT?
+ # self.data._close()
+ # self.unique_keys._close()
+ # self.searchable_keys._close()
+ pass
except BaseException:
pass
self.lock.release()
diff --git a/packages/syft/src/syft/types/blob_storage.py b/packages/syft/src/syft/types/blob_storage.py
index 8ef880dd6d4..d695bfd86d6 100644
--- a/packages/syft/src/syft/types/blob_storage.py
+++ b/packages/syft/src/syft/types/blob_storage.py
@@ -1,7 +1,12 @@
# stdlib
import mimetypes
from pathlib import Path
+from queue import Queue
import sys
+import threading
+from time import sleep
+from typing import Any
+from typing import ClassVar
from typing import List
from typing import Optional
from typing import Type
@@ -14,27 +19,131 @@
from ..node.credentials import SyftVerifyKey
from ..serde import serialize
from ..serde.serializable import serializable
+from ..service.action.action_object import ActionObject
+from ..service.action.action_object import BASE_PASSTHROUGH_ATTRS
+from ..service.action.action_types import action_types
from ..service.response import SyftException
+from ..service.service import from_api_or_context
+from ..types.transforms import drop
from ..types.transforms import keep
+from ..types.transforms import make_set_default
from ..types.transforms import transform
from .datetime import DateTime
+from .syft_migration import migrate
from .syft_object import SYFT_OBJECT_VERSION_1
+from .syft_object import SYFT_OBJECT_VERSION_2
from .syft_object import SyftObject
from .uid import UID
@serializable()
-class BlobFile(SyftObject):
+class BlobFileV1(SyftObject):
__canonical_name__ = "BlobFile"
__version__ = SYFT_OBJECT_VERSION_1
file_name: str
+ __repr_attrs__ = ["id", "file_name"]
+
+
+@serializable()
+class BlobFile(SyftObject):
+ __canonical_name__ = "BlobFile"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ file_name: str
+ syft_blob_storage_entry_id: Optional[UID] = None
+ file_size: Optional[int] = None
+
+ __repr_attrs__ = ["id", "file_name"]
+
+ def read(self, stream=False, chunk_size=512, force=False):
+ # get blob retrieval object from api + syft_blob_storage_entry_id
+ read_method = from_api_or_context(
+ "blob_storage.read", self.syft_node_location, self.syft_client_verify_key
+ )
+ blob_retrieval_object = read_method(self.syft_blob_storage_entry_id)
+ return blob_retrieval_object._read_data(stream=stream, chunk_size=chunk_size)
+
+ @classmethod
+ def upload_from_path(self, path, client):
+ # syft absolute
+ import syft as sy
+
+ return sy.ActionObject.from_path(path=path).send(client).syft_action_data
+
+ def _iter_lines(self, chunk_size=512):
+ """Synchronous version of the async iter_lines"""
+ return self.read(stream=True, chunk_size=chunk_size)
+
+ def read_queue(self, queue, chunk_size, progress=False, buffer_lines=10000):
+ total_read = 0
+ for _i, line in enumerate(self._iter_lines(chunk_size=chunk_size)):
+ line_size = len(line) + 1 # add byte for \n
+ if self.file_size is not None:
+ total_read = min(self.file_size, total_read + line_size)
+ else:
+ # naive way of doing this, max be 1 byte off because the last
+ # byte can also be a \n
+ total_read += line_size
+ if progress:
+ queue.put((total_read, line))
+ else:
+ queue.put(line)
+ while queue.qsize() > buffer_lines:
+ sleep(0.1)
+ # Put anything not a string at the end
+ queue.put(0)
+
+ def iter_lines(self, chunk_size=512, progress=False):
+ item_queue: Queue = Queue()
+ threading.Thread(
+ target=self.read_queue,
+ args=(item_queue, chunk_size, progress),
+ daemon=True,
+ ).start()
+ item = item_queue.get()
+ while item != 0:
+ yield item
+ item = item_queue.get()
+
+ def _coll_repr_(self):
+ return {"file_name": self.file_name}
+
+
+@migrate(BlobFile, BlobFileV1)
+def downgrade_blobfile_v2_to_v1():
+ return [
+ drop(["syft_blob_storage_entry_id", "file_size"]),
+ ]
+
+
+@migrate(BlobFileV1, BlobFile)
+def upgrade_blobfile_v1_to_v2():
+ return [
+ make_set_default("syft_blob_storage_entry_id", None),
+ make_set_default("file_size", None),
+ ]
+
class BlobFileType(type):
pass
+class BlobFileObjectPointer:
+ pass
+
+
+@serializable()
+class BlobFileObject(ActionObject):
+ __canonical_name__ = "BlobFileOBject"
+ __version__ = SYFT_OBJECT_VERSION_1
+
+ syft_internal_type: ClassVar[Type[Any]] = BlobFile
+ syft_pointer_type = BlobFileObjectPointer
+ syft_passthrough_attrs = BASE_PASSTHROUGH_ATTRS
+
+
@serializable()
class SecureFilePathLocation(SyftObject):
__canonical_name__ = "SecureFilePathLocation"
@@ -56,7 +165,7 @@ class SeaweedSecureFilePathLocation(SecureFilePathLocation):
@serializable()
-class BlobStorageEntry(SyftObject):
+class BlobStorageEntryV1(SyftObject):
__canonical_name__ = "BlobStorageEntry"
__version__ = SYFT_OBJECT_VERSION_1
@@ -68,9 +177,41 @@ class BlobStorageEntry(SyftObject):
uploaded_by: SyftVerifyKey
created_at: DateTime = DateTime.now()
+ __attr_searchable__ = ["bucket_name"]
+
@serializable()
-class BlobStorageMetadata(SyftObject):
+class BlobStorageEntry(SyftObject):
+ __canonical_name__ = "BlobStorageEntry"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ id: UID
+ location: Union[SecureFilePathLocation, SeaweedSecureFilePathLocation]
+ type_: Optional[Type]
+ mimetype: str = "bytes"
+ file_size: int
+ no_lines: Optional[int] = 0
+ uploaded_by: SyftVerifyKey
+ created_at: DateTime = DateTime.now()
+ bucket_name: Optional[str]
+
+ __attr_searchable__ = ["bucket_name"]
+
+
+@migrate(BlobStorageEntry, BlobStorageEntryV1)
+def downgrade_blobstorageentry_v2_to_v1():
+ return [
+ drop(["no_lines", "bucket_name"]),
+ ]
+
+
+@migrate(BlobStorageEntryV1, BlobStorageEntry)
+def upgrade_blobstorageentry_v1_to_v2():
+ return [make_set_default("no_lines", 1), make_set_default("bucket_name", None)]
+
+
+@serializable()
+class BlobStorageMetadataV1(SyftObject):
__canonical_name__ = "BlobStorageMetadata"
__version__ = SYFT_OBJECT_VERSION_1
@@ -79,6 +220,29 @@ class BlobStorageMetadata(SyftObject):
file_size: int
+@serializable()
+class BlobStorageMetadata(SyftObject):
+ __canonical_name__ = "BlobStorageMetadata"
+ __version__ = SYFT_OBJECT_VERSION_2
+
+ type_: Optional[Type[SyftObject]]
+ mimetype: str = "bytes"
+ file_size: int
+ no_lines: Optional[int] = 0
+
+
+@migrate(BlobStorageMetadata, BlobStorageMetadataV1)
+def downgrade_blobmeta_v2_to_v1():
+ return [
+ drop(["no_lines"]),
+ ]
+
+
+@migrate(BlobStorageMetadataV1, BlobStorageMetadata)
+def upgrade_blobmeta_v1_to_v2():
+ return [make_set_default("no_lines", 1)]
+
+
@serializable()
class CreateBlobStorageEntry(SyftObject):
__canonical_name__ = "CreateBlobStorageEntry"
@@ -103,6 +267,8 @@ def from_path(cls, fp: Union[str, Path], mimetype: Optional[str] = None) -> Self
if not path.is_file():
raise SyftException(f"{fp} is not a file.")
+ if fp.suffix.lower() == ".jsonl":
+ mimetype = "application/json-lines"
if mimetype is None:
mime_types = mimetypes.guess_type(fp)
if len(mime_types) > 0 and mime_types[0] is not None:
@@ -128,3 +294,6 @@ def file_name(self) -> str:
@transform(BlobStorageEntry, BlobStorageMetadata)
def storage_entry_to_metadata():
return [keep(["id", "type_", "mimetype", "file_size"])]
+
+
+action_types[BlobFile] = BlobFileObject
diff --git a/packages/syft/src/syft/types/syft_object.py b/packages/syft/src/syft/types/syft_object.py
index 42c8110536e..095f197f891 100644
--- a/packages/syft/src/syft/types/syft_object.py
+++ b/packages/syft/src/syft/types/syft_object.py
@@ -8,6 +8,7 @@
import inspect
from inspect import Signature
import re
+import traceback
import types
from typing import Any
from typing import Callable
@@ -778,7 +779,9 @@ def list_dict_repr_html(self) -> str:
)
except Exception as e:
- print(f"error representing {type(self)} of objects. {e}")
+ print(
+ f"error representing {type(self)} of objects. {e}, {traceback.format_exc()}"
+ )
pass
# stdlib
diff --git a/packages/syft/src/syft/types/transforms.py b/packages/syft/src/syft/types/transforms.py
index ae3200fbc38..01011f0a518 100644
--- a/packages/syft/src/syft/types/transforms.py
+++ b/packages/syft/src/syft/types/transforms.py
@@ -37,7 +37,10 @@ class TransformContext(Context):
def from_context(obj: Any, context: Optional[Context] = None) -> Self:
t_context = TransformContext()
t_context.obj = obj
- t_context.output = dict(obj)
+ try:
+ t_context.output = dict(obj)
+ except Exception:
+ t_context.output = obj.to_dict()
if hasattr(context, "credentials"):
t_context.credentials = context.credentials
if hasattr(context, "node"):
diff --git a/packages/syft/src/syft/util/util.py b/packages/syft/src/syft/util/util.py
index 9a464d8c5ef..6cf80a3e9c4 100644
--- a/packages/syft/src/syft/util/util.py
+++ b/packages/syft/src/syft/util/util.py
@@ -872,6 +872,10 @@ def thread_ident() -> int:
return threading.current_thread().ident
+def proc_id() -> int:
+ return os.getpid()
+
+
def set_klass_module_to_syft(klass, module_name):
if module_name not in sys.modules["syft"].__dict__:
new_module = types.ModuleType(module_name)
diff --git a/packages/syft/tests/syft/service/jobs/job_stash_test.py b/packages/syft/tests/syft/service/jobs/job_stash_test.py
new file mode 100644
index 00000000000..a228bea6f2e
--- /dev/null
+++ b/packages/syft/tests/syft/service/jobs/job_stash_test.py
@@ -0,0 +1,45 @@
+# stdlib
+from datetime import datetime
+from datetime import timedelta
+
+# third party
+import pytest
+
+# syft absolute
+from syft.service.job.job_stash import Job
+from syft.service.job.job_stash import JobStatus
+from syft.types.uid import UID
+
+
+@pytest.mark.parametrize(
+ "current_iter, n_iters, status, creation_time_delta, expected",
+ [
+ (0, 10, JobStatus.CREATED, timedelta(hours=2), None),
+ (1, None, JobStatus.CREATED, timedelta(hours=2), None),
+ (5, 10, JobStatus.PROCESSING, timedelta(hours=2), "24:00s/it"),
+ (200000, 200000, JobStatus.COMPLETED, timedelta(hours=2), "<00:00"),
+ (156000, 200000, JobStatus.PROCESSING, timedelta(hours=2), "00:00s/it"),
+ (1, 3, JobStatus.PROCESSING, timedelta(hours=2), "2:00:00s/it"),
+ (10, 10, JobStatus.PROCESSING, timedelta(minutes=5), "00:30s/it"),
+ (0, 10, JobStatus.CREATED, timedelta(days=1), None),
+ (10, 100, JobStatus.PROCESSING, timedelta(seconds=3600), "06:00s/it"),
+ (100000, 200000, JobStatus.PROCESSING, timedelta(minutes=1), "00:00s/it"),
+ (2, 10, JobStatus.PROCESSING, timedelta(seconds=119.6), "00:59s/it"),
+ ],
+)
+def test_eta_string(current_iter, n_iters, status, creation_time_delta, expected):
+ job = Job(
+ id=UID(),
+ node_uid=UID(),
+ n_iters=n_iters,
+ current_iter=current_iter,
+ creation_time=(datetime.now() - creation_time_delta).isoformat(),
+ status=status,
+ )
+
+ if expected is None:
+ assert job.eta_string is None
+ else:
+ assert job.eta_string is not None
+ assert isinstance(job.eta_string, str)
+ assert expected in job.eta_string
diff --git a/packages/syft/tests/syft/stores/action_store_test.py b/packages/syft/tests/syft/stores/action_store_test.py
index 853058ae3f5..0994d2ae168 100644
--- a/packages/syft/tests/syft/stores/action_store_test.py
+++ b/packages/syft/tests/syft/stores/action_store_test.py
@@ -1,4 +1,5 @@
# stdlib
+import sys
from typing import Any
# third party
@@ -53,6 +54,7 @@ def test_action_store_sanity(store: Any):
)
@pytest.mark.parametrize("permission", permissions)
@pytest.mark.flaky(reruns=3, reruns_delay=1)
+@pytest.mark.skipif(sys.platform == "darwin", reason="skip on mac")
def test_action_store_test_permissions(store: Any, permission: Any):
client_key = SyftVerifyKey.from_string(test_verify_key_string_client)
root_key = SyftVerifyKey.from_string(test_verify_key_string_root)
diff --git a/packages/syft/tests/syft/syft_functions/__init__.py b/packages/syft/tests/syft/syft_functions/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/packages/syft/tests/syft/syft_functions/syft_function_test.py b/packages/syft/tests/syft/syft_functions/syft_function_test.py
new file mode 100644
index 00000000000..5b53886eaf7
--- /dev/null
+++ b/packages/syft/tests/syft/syft_functions/syft_function_test.py
@@ -0,0 +1,96 @@
+# stdlib
+import random
+import sys
+from textwrap import dedent
+
+# third party
+import pytest
+
+# syft absolute
+import syft as sy
+from syft import ActionObject
+from syft import syft_function
+from syft import syft_function_single_use
+from syft.service.response import SyftError
+from syft.service.response import SyftSuccess
+
+
+@pytest.fixture
+def node():
+ _node = sy.orchestra.launch(
+ name="nested_job_test_domain",
+ dev_mode=True,
+ reset=True,
+ n_consumers=3,
+ create_producer=True,
+ queue_port=random.randint(13000, 13300),
+ )
+ # startup code here
+ yield _node
+ # Cleanup code
+ _node.land()
+
+
+@pytest.mark.flaky(reruns=5, reruns_delay=1)
+@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
+def test_nested_jobs(node):
+ client = node.login(email="info@openmined.org", password="changethis")
+
+ res = client.register(name="a", email="aa@b.org", password="c", password_verify="c")
+ assert isinstance(res, SyftSuccess)
+ ds_client = node.login(email="aa@b.org", password="c")
+ ## Dataset
+
+ x = ActionObject.from_obj([1, 2])
+ x_ptr = x.send(ds_client)
+
+ ## aggregate function
+ @sy.syft_function()
+ def aggregate_job(job_results):
+ return sum(job_results)
+
+ aggregate_job.code = dedent(aggregate_job.code)
+ res = ds_client.code.submit(aggregate_job)
+
+ ## Batch function
+ @syft_function()
+ def process_batch(batch):
+ print(f"starting batch {batch}")
+ return batch + 1
+
+ process_batch.code = dedent(process_batch.code)
+
+ res = ds_client.code.submit(process_batch)
+ print(res)
+
+ ## Main function
+
+ @syft_function_single_use(x=x_ptr)
+ def process_all(domain, x):
+ job_results = []
+ for elem in x:
+ batch_job = domain.launch_job(process_batch, batch=elem)
+ job_results += [batch_job.result]
+
+ result = domain.launch_job(aggregate_job, job_results=job_results)
+ return result.wait().get()
+
+ process_all.code = dedent(process_all.code)
+
+ # Approve & run
+ res = ds_client.code.request_code_execution(process_all)
+ print(res)
+ assert not isinstance(res, SyftError)
+ client.requests[-1].approve(approve_nested=True)
+
+ job = ds_client.code.process_all(x=x_ptr, blocking=False)
+
+ job.wait()
+ # stdlib
+
+ assert len(job.subjobs) == 3
+ # stdlib
+
+ sub_results = [j.wait().get() for j in job.subjobs]
+ assert set(sub_results) == {2, 3, 5}
+ assert job.wait().get() == 5
diff --git a/packages/syft/tests/syft/users/user_code_test.py b/packages/syft/tests/syft/users/user_code_test.py
index 9fe191e25ea..6c99e63869d 100644
--- a/packages/syft/tests/syft/users/user_code_test.py
+++ b/packages/syft/tests/syft/users/user_code_test.py
@@ -84,3 +84,37 @@ def func(asset):
)
assert status_change.linked_obj.resolve.assets[0] == asset_input
+
+
+@sy.syft_function()
+def test_inner_func():
+ return 1
+
+
+@sy.syft_function(
+ input_policy=sy.ExactMatch(), output_policy=sy.SingleExecutionExactOutput()
+)
+def test_outer_func(domain):
+ job = domain.launch_job(test_inner_func)
+ return job
+
+
+def test_nested_requests(worker, guest_client: User):
+ guest_client.api.services.code.submit(test_inner_func)
+ guest_client.api.services.code.request_code_execution(test_outer_func)
+
+ root_domain_client = worker.root_client
+ request = root_domain_client.requests[-1]
+ assert request.code.nested_requests == {"test_inner_func": "latest"}
+ root_domain_client.api.services.request.apply(request.id)
+ request = root_domain_client.requests[-1]
+
+ codes = root_domain_client.code
+ inner = codes[0] if codes[0].service_func_name == "test_inner_func" else codes[1]
+ outer = codes[0] if codes[0].service_func_name == "test_outer_func" else codes[1]
+ assert list(request.code.nested_codes.keys()) == ["test_inner_func"]
+ (linked_obj, node) = request.code.nested_codes["test_inner_func"]
+ assert node == {}
+ assert linked_obj.resolve.id == inner.id
+ assert outer.status.approved
+ assert not inner.status.approved
diff --git a/packages/syft/tests/syft/users/user_test.py b/packages/syft/tests/syft/users/user_test.py
index 3b56ca54a9e..b5743effd4f 100644
--- a/packages/syft/tests/syft/users/user_test.py
+++ b/packages/syft/tests/syft/users/user_test.py
@@ -39,9 +39,10 @@ def get_mock_client(root_client, role) -> DomainClient:
mail = Faker().email()
name = Faker().name()
password = "pw"
- assert root_client.register(
+ user = root_client.register(
name=name, email=mail, password=password, password_verify=password
)
+ assert user
user_id = [u for u in get_users(worker) if u.email == mail][0].id
assert worker.root_client.api.services.user.update(
user_id, UserUpdate(user_id=user_id, role=role)
diff --git a/packages/syft/tests/syft/zmq_queue_test.py b/packages/syft/tests/syft/zmq_queue_test.py
index 2439ca0acc8..50d7d1ac733 100644
--- a/packages/syft/tests/syft/zmq_queue_test.py
+++ b/packages/syft/tests/syft/zmq_queue_test.py
@@ -1,9 +1,12 @@
# stdlib
from collections import defaultdict
import random
+import sys
+from time import sleep
# third party
from faker import Faker
+import pytest
from zmq import Socket
# syft absolute
@@ -19,14 +22,22 @@
from syft.service.response import SyftSuccess
-def test_zmq_client():
+@pytest.fixture
+def client():
hostname = "127.0.0.1"
-
config = ZMQClientConfig(hostname=hostname)
+ client = ZMQClient(config=config)
+ yield client
+ # Cleanup code
+ client.close()
- assert config.hostname == hostname
- client = ZMQClient(config=config)
+@pytest.mark.flaky(reruns=5, reruns_delay=1)
+@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
+def test_zmq_client(client):
+ hostname = "127.0.0.1"
+
+ assert client.config.hostname == hostname
assert client.host == hostname
assert len(client.producers) == 0
@@ -58,6 +69,12 @@ def handle_message(message: bytes):
consumer = client.add_consumer(
queue_name=QueueName, message_handler=MyMessageHandler
)
+
+ consumer.run()
+ # stdlib
+ from time import sleep
+
+ sleep(1)
assert isinstance(consumer, ZMQConsumer)
assert consumer.address is not None
assert consumer.alive
@@ -70,67 +87,97 @@ def handle_message(message: bytes):
assert QueueName in client.consumers
assert len(client.consumers[QueueName]) > 0
- response = client.send_message(message=b"My Message", queue_name=QueueName)
+ msg = [producer.identity, b"", b"My Message"]
+ response = client.send_message(
+ message=msg, queue_name=QueueName, worker=consumer.identity
+ )
assert isinstance(response, SyftSuccess)
- consumer.receive()
+ sleep(0.5)
+ # consumer.receive()
assert len(received_message) == 1
- response = client.send_message(message="My Message", queue_name="random queue")
+ msg = [producer.identity, b"", b"My Message"]
+ response = client.send_message(message=msg, queue_name="random queue")
assert isinstance(response, SyftError)
assert isinstance(client.close(), SyftSuccess)
+ sleep(0.5)
assert client.producers[QueueName].alive is False
assert client.consumers[QueueName][0].alive is False
-def test_zmq_pub_sub(faker: Faker):
- received_messages = []
+@pytest.fixture
+def producer():
+ pub_port = random.randint(11000, 12000)
+ QueueName = "ABC"
- pub_port = random.randint(6001, 10004)
+ # Create a producer
+ producer = ZMQProducer(
+ port=pub_port, queue_name=QueueName, queue_stash=None, context=None
+ )
+ yield producer
+ # Cleanup code
+ if producer.alive:
+ producer.close()
- pub_addr = f"tcp://127.0.0.1:{pub_port}"
- QueueName = "ABC"
+@pytest.fixture
+def consumer(producer):
+ # Create a consumer
+ consumer = ZMQConsumer(
+ message_handler=None,
+ address=producer.address,
+ queue_name=producer.queue_name,
+ )
+ yield consumer
+ # Cleanup code
+ if consumer.alive:
+ consumer.close()
- # Create a producer
- producer = ZMQProducer(address=pub_addr, queue_name=QueueName)
+
+@pytest.mark.flaky(reruns=5, reruns_delay=1)
+@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
+def test_zmq_pub_sub(faker: Faker, producer, consumer):
+ received_messages = []
+
+ pub_addr = f"tcp://localhost:{producer.port}"
assert producer.address == pub_addr
- assert isinstance(producer._producer, Socket)
+ assert isinstance(producer.backend, Socket)
assert isinstance(producer, ZMQProducer)
- assert producer.queue_name == QueueName
assert producer.alive
+ queue_name = producer.queue_name
+
first_message = faker.sentence().encode()
class MyMessageHandler(AbstractMessageHandler):
- queue = QueueName
+ queue = producer.queue_name
@staticmethod
def handle_message(message: bytes):
received_messages.append(message)
- # Create a consumer
- consumer = ZMQConsumer(
- message_handler=MyMessageHandler,
- address=pub_addr,
- queue_name=QueueName,
- )
+ consumer.message_handler = MyMessageHandler
assert isinstance(consumer, ZMQConsumer)
assert consumer.address == pub_addr
- assert isinstance(consumer._consumer, Socket)
- assert consumer.queue_name == QueueName
+ assert isinstance(consumer.worker, Socket)
+ assert consumer.queue_name == queue_name
assert consumer.alive
assert consumer.thread is None
assert consumer.message_handler == MyMessageHandler
+ consumer.run()
+ sleep(0.2)
- producer.send(message=first_message)
+ msg = [producer.identity, b"", first_message]
+ producer.send(message=msg, worker=consumer.identity)
# Check if consumer receives the message
- consumer.receive()
+ # consumer.receive()
+ sleep(0.2)
# Validate if message was correctly received in the handler
assert len(received_messages) == 1
@@ -143,14 +190,24 @@ def handle_message(message: bytes):
assert consumer.alive is False
-def test_zmq_queue_manager() -> None:
+@pytest.fixture
+def queue_manager():
+ # Create a consumer
config = ZMQQueueConfig()
+ queue_manager = QueueManager(config=config)
+ yield queue_manager
+ # Cleanup code
+ queue_manager.close()
+
+
+@pytest.mark.flaky(reruns=5, reruns_delay=1)
+@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
+def test_zmq_queue_manager(queue_manager) -> None:
+ config = queue_manager.config
assert isinstance(config.client_config, ZMQClientConfig)
assert config.client_type == ZMQClient
- queue_manager = QueueManager(config=config)
-
assert queue_manager.client_config.hostname
assert isinstance(queue_manager._client, ZMQClient)
@@ -169,11 +226,15 @@ class CustomHandler(AbstractMessageHandler):
def handle_message(message: bytes):
received_messages.append(message)
- producer = queue_manager.create_producer(queue_name=QueueName)
+ producer = queue_manager.create_producer(
+ queue_name=QueueName, queue_stash=None, context=None
+ )
assert isinstance(producer, ZMQProducer)
- consumer = queue_manager.create_consumer(message_handler=CustomHandler)
+ consumer = queue_manager.create_consumer(
+ message_handler=CustomHandler, address=producer.address
+ )
assert isinstance(consumer, ZMQConsumer)
diff --git a/scripts/print_fd.py b/scripts/print_fd.py
new file mode 100644
index 00000000000..4d9944dfc10
--- /dev/null
+++ b/scripts/print_fd.py
@@ -0,0 +1,104 @@
+# stdlib
+import argparse
+from collections import defaultdict
+import subprocess
+
+
+def run_lsof():
+ """Run the lsof command and return its output."""
+ try:
+ process = subprocess.Popen(["lsof"], stdout=subprocess.PIPE, text=True)
+ output, _ = process.communicate()
+ return output
+ except Exception as e:
+ print(f"Error running lsof: {e}")
+ return ""
+
+
+def run_lsof_for_pid(pid):
+ """Run the lsof command for a specific PID and return its output."""
+ try:
+ process = subprocess.Popen(
+ ["lsof", "-p", str(pid)], stdout=subprocess.PIPE, text=True
+ )
+ output, _ = process.communicate()
+ return output
+ except Exception as e:
+ print(f"Error running lsof for PID {pid}: {e}")
+ return ""
+
+
+def parse_lsof_output(lsof_output, verbose):
+ """Parse the lsof output."""
+ data = defaultdict(list)
+ lines = lsof_output.splitlines()
+
+ for line in lines[1:]: # Skip header line
+ parts = line.split(maxsplit=8)
+ if len(parts) < 9 or "python" not in parts[0].lower():
+ continue # Skip lines that are not Python processes
+
+ proc_name, pid, owner, fd_type, fd_info, _, _, _, file_path = parts
+ # Skip site-packages paths if not in verbose mode
+ filters = [
+ "site-packages",
+ "lib-dynload",
+ "cellar",
+ ".pyenv",
+ "ttys",
+ "/dev/null",
+ "/dev/random",
+ "/dev/urandom",
+ "localhost",
+ ]
+ skip = False
+ if not verbose:
+ for filter in filters:
+ if filter in file_path.lower():
+ skip = True
+ break
+ if skip:
+ continue
+
+ data[pid].append(
+ {
+ "Owner": owner,
+ "FD Type": fd_type,
+ "FD Info": fd_info,
+ "File Path": file_path,
+ }
+ )
+
+ return data
+
+
+def main(pid=None, verbose=False):
+ lsof_output = run_lsof_for_pid(pid) if pid else run_lsof()
+ files_by_pid = parse_lsof_output(lsof_output, verbose)
+
+ for pid, files in files_by_pid.items():
+ print(f"PID {pid} open files:")
+ for file in files:
+ print(f" {file['File Path']} ({file['FD Type']} - {file['FD Info']})")
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(
+ description="List open files for Python processes."
+ )
+ parser.add_argument(
+ "pid",
+ nargs="?",
+ type=int,
+ default=None,
+ help="The PID of the Python process (optional).",
+ )
+ parser.add_argument(
+ "-v",
+ "--verbose",
+ action="store_true",
+ help="Include all file descriptors, including those in site-packages.",
+ )
+ args = parser.parse_args()
+
+ main(args.pid, args.verbose)
diff --git a/tox.ini b/tox.ini
index 726e9fdcdd9..1467f7f52b4 100644
--- a/tox.ini
+++ b/tox.ini
@@ -416,11 +416,14 @@ description = Syft Unit Tests
deps =
{[testenv:syft]deps}
{[testenv:hagrid]deps}
+allowlist_externals =
+ bash
changedir = {toxinidir}/packages/syft
setenv =
ENABLE_SIGNUP=False
commands =
pip list
+ bash -c 'ulimit -n 4096 || true'
pytest -n auto
[testenv:stack.test.integration.enclave.oblv]