-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to cross-compile cryptography with rust support? #5814
Comments
Can you try setting the env var |
The error message is less puzzling, but it crashed at the same place:
|
Looks like this is a platform limitation: https://users.rust-lang.org/t/target-aarch64-apple-ios-does-not-support-these-cdylib-crate-type/55255 That thread suggests that no |
Thanks. I've read the post you link. That is a very dated view; I am definitely using dynamic libraries in iOS (I have to convert them to frameworks, but that's a straightforward process; I am already doing it with |
I think the answer here is simply "Rust's view of what's possible on iOS is
out of date, and you should file an issue with rustc asking them to fix
this" :-) If you file an issue please do link it here!
…On Thu, Feb 11, 2021 at 12:41 PM Nicolas Holzschuch < ***@***.***> wrote:
Thanks. I've read the post you link. That is a very dated view; I am
definitely using dynamic libraries in iOS (I have to convert them to
frameworks, but that's a straightforward process; I am already doing it
with cryptography/hazmat/bindings/_padding.abi3.so and
cryptography/hazmat/bindings/_openssl.abi3.so). I'll keep investigating.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#5814 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBED6LLZC37HMCQOWMTS6QJF5ANCNFSM4XPG7EPA>
.
--
All that is necessary for evil to succeed is for good people to do nothing.
|
Done. That issue has been open since 2015, though. |
FYI I'm in the process of porting the rust setuptools infrastructure to buildroot, this is roughly what my package makefile looks like:
Note that this currently depends on the following unmerged patches: I should add that the rust requirement appears somewhat likely to cause serious security issues for embedded Linux if there is no long term support plan for the many architectures without full rust toolchain support as rust toolchain support is currently extremely limited compared to our supported architectures. I would highly recommend some sort of longer term migration plan such as a LTS branch or non-rust fallbacks until the rust toolchain support matures more. I do think moving to rust long term will be beneficial to security, however for a critical library like cryptography making rust mandatory without a sufficiently long migration period has the potential to be highly counterproductive as the likely result will be pinning old unsupported versions. In any case it's not a major issue for any of my projects using buildroot as those do happen to have rust full toolchain support but I can see it being a serious issue for others. I think compared to the gentoo issue the issues with rust cross compilation might be even more severe due to the toolchain bootstrapping issues(which from my understanding is why buildroot only supports rust for platforms with prebuilt toolchains). Normally buildroot bootstraps all required cross toolchains by building them from source, rust is basically the only exception there from my understanding. |
Personally I consider Rust as a litmus test for platforms. Rust 1.0 was released more than half a decade ago. It is like that any platform without Rust support simply does not attract sufficient attention from researcher and developers. Which by itself commonly means that these platforms have a small user base. OSS projects have limit resources and have to make compromises how do spend their resources. To quote Spock: "The needs of the many outweigh the needs of the few" I recommend that your work with Rust team and GCC's gccrs team to improve support for your platforms instead. |
Embedded systems in my experience are quite different there, you will often(most of the time probably) have a very large user base that doesn't even know what OS they are running. For example, how many users even know their home router is likely Linux based, and how many users have a Linux based router, one is vastly larger than the other. And exotic architectures tend to be relatively common for embedded systems. It's also difficult as is getting vendors to stay up to date with security, throw in a bunch more toolchain issues(on top of the ones they already are fighting half the time) and the issues are likely to get worse.
Sure, I'm just pointing out that if the decision is being made based on the assumption that the vast majority of users(which is quite different from developers) have a rust supporting platform, then there's a good chance the current situation is quite a bit different than assumed, simply due to a library like cryptography being part of a huge amount of infrastructure software that often isn't very visible.
Sure, but that's not something that will really help much in the short term, things are improving and I'm doing what I can to help but these things take time and I think for a change this big it would benefit users a lot to have a longer migration period, even if a non-rust version would be just receiving security fixes only. At least that would mitigate the bulk of the near term security issues on the embedded Linux side of things. The gccrs port looks interesting, seems like it could bring rust to a lot more architectures a lot easier than trying to port LLVM, I'm not really a compiler developer myself but I'll def try and help with packaging and testing once it gets more mature. |
(Disclaimer: I'm not a core developer of PyCA cryptography, only a contributor)
Your words could be interpreted as guilt-pressuring unpaid Open Source developers into supporting these embedded platforms, as otherwise "humans may die a horrible death when embedded controller of an elevator fails". A bit unfair, don't you think? </exaggeration> I got your point of view. The decision to use Rust puts your in an uncomfortable position. You want to archive what's best for your work, product and customers. Please also understand that PyCA cryptography is a literally a two-man show. Alex and Paul are doing 99% of all work. I'm honestly worried that this week's shit storm and following peer pressure is going to burn them out. Ultimately security updates and software maintenance is a social, legal, and political issue that is out of scope for a little OSS project as PyCA cryptography. For example the EU is working on laws in regards of right-to-repair and mandatory security updates during the life cycle of a product. |
This is an unfortunate situation, but it's also unfair to ask others to shoulder the burden of support which companies are unwilling to appropriately invest in. Domains like embedded systems push the externalities of supporting them on everyone else. Some of that is inevitable, but it's also important to keep pushing forward the state of our industry even if it results in some pain. We would love for Rust (and our project) to be available everywhere! However, ubiquity is created through demand. In most of the cases we've seen so far the challenge has been about getting the right people talking to each other instead of any specific technical obstacle. This makes me hopeful that for many of these platforms the problem will be solved sooner rather than later. We also had an LTS discussion in #5799. The outcome was that while we are not going to explicitly maintain an LTS (see the thread for why this is such a commitment), we've made a small change to how we handle CVEs to make backports easier for packagers (#5799 (comment)). This should make the runway longer for those who absolutely need it. |
It's not an issue for me personally, since the platforms I currently support(x86_64 and aarch64) have rust cross toolchains, but I noticed the extremely limited rust toolchain architecture support when working on bringing up setuptools-rust for buildroot.
I get that, but one should realize that this is a hugely disruptive change for packagers/integrators. Maybe I'm underestimating the maintenance burden of providing better migration paths here but I find backwards compat to be extremely important for users of infrastructure projects like this, mostly since I've seen what usually happens when updating a dependency becomes difficult(it either won't get updated or will be replaced with a different dependency/fork). Generally fixing CVE's in a LTS branch as they come in is not particularly time consuming unless there are serious fundamental design flaws that require reworking. Maybe some of the distro maintainers would be willing to help maintain a backwards compatible version.
IMO a big reason embedded systems have these security update issues is due to infrastructure projects in their dependency trees not providing good migration paths, I'd say this falls under the social issue there. The Linux kernel has been improving here a lot at least with extended LTS support periods but the userspace situation is not very good overall.
I'd say for the most part the reality of the situation is the opposite, embedded systems manufacturers typically handle most maintenance/updates themselves, which means they will typically just version pin or hack around dependencies that have problematic updates, sure some work with their upstreams but that's generally not the most common case. I usually find it easier to push things forward by providing better migration paths, just look at the python 2 to python 3 migration, which didn't seriously take off until features/shims that made single source universal python2/python3 code workable were introduced.
I agree that rust should be available in more places, but this is a long tail sort of issue due to the extremely extended support periods for many platforms/architectures and toolchain complexity. Supporting rust on a lot of these platforms is essentially an unsolvable issue in the near term unfortunately. |
Now that buildroot has merged most of the patches I sent in adding the rust setuptools build infrastructure needed for building python-cryptography I've now sent in the patch which updates cryptography here: |
I'm running into some bad longstanding bugs/missing config options in regards to how rust is handling cross toolchain build script linking that seems to be blocking cryptography cross compilation from being workable for upstream buildroot at the moment even for normal x86_64 targets. |
@jameshilliard You might also be interested in rust-lang/cargo#8147. I have been doing a lot of what I would want to claim is "simple" cross-compiling with Rust over the past year and a half--notably, I am not even working with niche targets!--but they seem to only particularly care about a handful of specific host/target configurations (and even some of those are a bit broken, such as macOS -> iOS)... host==target has never gotten any real interest, sadly. I hope you make progress on this, as my workarounds are getting more and more hilarious (I have been known to just tell Rust "my linker is /bin/true" so I can get it to stop messing me up ;P). |
Yeah, I could somewhat understand if there was a viable workaround for this but I have yet to find anything that looks even remotely maintainable, we maintain a lot of cross compilation workarounds/hacks in buildroot but this situation seems totally ridiculous to me.
Maybe with the Linux kernel working on a rust integration the situation might improve...I really had no idea rust cross compilation was this immature/broken when I started working on adding python rust support to buildroot. Best I can tell the main reason this is still broken seems to be bike-shedding of the issue endlessly. I'm not familiar with the rust/cargo codebase but if I had to guess it's probably only a few lines of code to add a build script linker override env variable or something that would at least let us override this broken behavior in some way. At a certain point it becomes more important to just provide a way for others to make it work in some way rather than to try and bike-shed the most ideal solution. Normally I could use the buildroot patch system to workaround this but since it's not really all that viable to build a rust toolchain from source due to the rust requires rust to build bootstrapping issue...that's not even workable. |
So for a status update on getting these target==host architecture cross compilation bugs fixed, the pyo3 fix has been merged in #1514, in addition I have a pending pull request fixing the related cargo bug in #9322. |
Thank you for your work on this!
…On Mon, Apr 5, 2021 at 2:18 PM James Hilliard ***@***.***> wrote:
I'm running into some bad longstanding bugs/missing config options in
regards to how rust is handling cross toolchain build script linking that
seems to be blocking cryptography cross compilation from being workable for
upstream buildroot at the moment even for normal x86_64 targets.
See:
#3349 <rust-lang/cargo#3349>
#1513 <PyO3/pyo3#1513>
So for a status update on getting these target==host architecture cross
compilation bugs fixed, the pyo3 fix has been merged in #1514
<PyO3/pyo3#1514>, in addition I have a pending
pull request fixing the related cargo bug in #9322
<rust-lang/cargo#9322>.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#5814 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBCVO5TQQQPXDSEOBXTTHH5HHANCNFSM4XPG7EPA>
.
--
All that is necessary for evil to succeed is for good people to do nothing.
|
So cross compilation seems to mostly work on nightly rust if you set In general rust cross-compilation seems to be pretty marginal even with that fix but at least it's making progress...a lot of the rust cross compilation tooling and meta buildsystem integrations(ie buildroot) are fairly immature still though so I would expect a lot of tooling bugs until things stabilize a bit more. |
Looks like pyo3 just had a release( |
Based on the latest comments indicating that this seems to be working, I'm going to close this. Thanks to everyone who helped on the upstream work. |
Hi,
I'm trying to compile the new version of cryptography, with rust support, for iOS and iOS simulator, on a Mac.
Cross-compiling the C files works by setting CFLAGS and CC inside the environment:
(I'll admit the command line is a bit scary, but it's basically just setting flags).
However, setting flags for rustc does not work in the current release. I've tried adding
RUSTFLAGS="--target=aarch64-apple-ios"
to the previous command line, but all I get is an error:As you can see, the
target
in the error message is not the target specified on the--target
flag, which is confusing. Adding the "--target" flag manually to the compile command works for a while longer:My question is: what's the recommended method to cross-compile cryptography, and how to set flags for the rust compiler?
The text was updated successfully, but these errors were encountered: