Add or-tools package

[BastianZim]

Checklist

• Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml".
• License file is packaged (see here for an example).
• Source is from official source.
• Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged).
• If static libraries are linked in, the license of the static library is packaged.
• Build number is 0.
• A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details).
• GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there.
• When in trouble, please check our knowledge base documentation before pinging a team.

Closes #2717

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/or-tools) and found it was in an excellent condition.

[BastianZim]

Also, happy for you to push to my branch directly. You should have the rights through being a maintainer but let me know if you need more.

[BastianZim]

Thanks for the review @xhochy. As both parts were copied from @wolfv, I'd leave that to him.

[wolfv]

I think we should try to remove the patch + the CXX mods.

[BastianZim]

Has been removed, let's see. @wolfv When you have a minute, would you mind having a look at my first comment above?

Edit: I tested it locally and it works when removing the BUILD_DEPS part. It just fails with the overlinking error.

[Mizux]

My 2 cents,

For or-tools we provide 3 build systems (bazel (c++ only), Make and CMake).
According to your trace, it seems you mess up CMake and Make build command...
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=375512&view=logs&j=6f142865-96c3-535c-b7ea-873d86b887bd&t=22b0682d-ab9e-55d7-9c79-49f3c3ba4823&l=917
note: until 917 seems to be a cmake configure stuff, then later it's using our usual Makefile (I mean using the Makefile based build not the Makefile generated by CMake...)
note: since we have a Makefile in the root directory you should try to NOT build in the source tree aka cmake -Bbuild (maybe the case since we can see some $SRC_DIR/UILD_DEPS=OFF note: strange name for a build dir ?)

note: you can come on our discord if you need help (see badge on the readme on github)

[BastianZim]

@wolfv By the way, if you’re too busy with Packaging-Con and everything else I‘m also happy to ask core to review it. Just let me know what you prefer. :)

[wolfv]

I am havign a look right now!

[wolfv]

1. dependencies look fine. The only problem will be the absl and protobuf pinning because we don't use the same as the rest of conda-forge. Hopefully in the future we might be able to change that.
2. Don't think we need additional C++ tests, it's fine (building the python bindings is a C++ test ;)
3. We can discuss additional solvers on the feedstock once created. Not sure if there is a pluggable way to enable other solvers. In that case having additional pacakges that just ship with the additional solver bits would be awesome.
4. Regarding the naming, I do think that or-tools vs ortools is pretty confusing. We could also just use ortools for python and ortools-cpp for C++ (like we do with boost and others). But if you and the or-tools community is agreeing on this (cc @Mizux) then we can do it like this.
5. OK let's wait :) It's always easy to iterate on the feedstock
6. Looks like we have description now in all three packages, I think that's fine. Unfortunately I'd also have to google regarding line breaks.
7. As long as it's properly "unpacked" and installed into the $PREFIX and we can import it in the test it should be fine. Might be nice to make sure it's dynamically linking to or-tools and all other libraries.
8. Sure!
9. It would probably good to add the version requirements with x.x right for the coin packages, right?
10. Done!
11. to be honest, I have no idea :)

Awesome job, I am happy to merge right away! Although I do believe that a name change to make the two packages more explicitly separated could be nice.

[Mizux]

*1) What is your process to integrate absl and protobuf ? i.e, How did you choose your version ? FYI before each google/or-tools release we try to bump our dependencies to use the last available release (note: I'm not a conda-forge user so don't know your policies, sorry).
*3) Currently most optionnal solvers are integrated using their C++ header + static lib (.a) so there are compile time optional dependencies, theo nly exception is gurobi which is using a runtime dynamic loading, we could do the same for all other otional solver (aka proprietary one we can't integrate by default) but it would need dev ressources we currently don't have.
*4) FYI In CMake the C++ target should be ortools::ortools so while the repo is google/or-tools it could make sense to call the package ortools-cpp or second option: ortools (the C++ package) and ortools-python (the python wrapper which could depends on ortools package in a near future)
*12) For coin-or it seems or-tools is working again with the clp stable-1.17 branch aka once coin-or release a release/1.17.7 tag or-tools should be able to compile against it (i.e. no more need of pinning IMHO...)

[BastianZim]

Thanks, @wolfv!

1. Packages
   1. abseil_cpp should be fine: https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/222a74dfdf8290be07572eea8850a0eef9f671c8/recipe/conda_build_config.yaml#L323
   2. protobuf is indeed a problem. Should we try updating protobuf or fix it here with ignore_run_exports? @Mizux do you know if or-tools could also be used with protobuf 3.16?
2. Ok
3. If you have Discord, I asked this question here: https://discordapp.com/channels/693088862481678374/693088862909759519/900065931429019660
   In short: Except for the gurobi solver, ortools has to be built for each solver individually for now. It's fairly simple though to add additional solvers because we only need to see a flag and build it again, so just copy/paste and change the name. I would maybe suggest that we open an issue at the feedstock so that everybody is aware that they can just make a PR if they need an additional solver. Options are here: https://github.com/google/or-tools/tree/stable/cmake#cmake-options
4. Agreed, I would follow with what @Mizux just posted and rename everything to the respective language (ortools-cpp, ortools-python) since we can also add java, etc. at a later point.
5. I think everybody in that PR is quite busy so maybe let's merge this and add OSX later, as you suggested?
6. Ok then I'll just keep it in for now.

7. Might be nice to make sure it's dynamically linking to or-tools and all other libraries.
   Would you mind doing that because, tbh, that's above my abilities. :)

8. Great!
9. Yes, definitely. Although we need to make sure to use coin-or-clp 1.17.4 because there is a bug somewhere with the newer versions.
   By the way, they all run_export now to x.x instead of >=x.x.x, just fyi. Any other dependencies that we should add?
10. 👍
11. Haha ok will ask Gitter.

Edit: @Mizux was faster than me, will answer that now. :)

[BastianZim]

Thanks @Mizux

1. Conda-forge has a list of globally pinned packages. This prevents one package from building against 1.1.2 of a package and another package from building against 1.1.3 and thus creating islands of incompatible packages. Some more info is here: https://conda-forge.org/docs/maintainer/pinning_deps.html#globally-pinned-packages
2. Thanks for the explanation. By the way, I completely forgot about BUILD_FLATZINC and BUILD_GLOP. We probably have to turn those off as well, right? I cannot find an option USE_GLOP though, so are they separate and just supplied as a convenience?
3. Ok will rename.
4. Ok that's great news so does that mean that 1.17.5 and 1.17.6 have to be skipped or do all of them work again?

[Mizux]

*1) For protobuf 1.16 it may work (AFAIK we don't use new feature and our proto are relatively stable, should read the protobuf 3.16 changelog) BUT currently we don't have job and resources to test a range of version to see what works and what not -> so I would say you can test and see what happen.
EDIT: IIRC sat Python wrapper is basically based on creating a proto file then passing it to the C++ layer so if python sat samples pass you should be green ;)
EDIT2: an other issue some users have is tensorflow python module also import protobuf so to be able to use or-tools AND tensorflow in the same script you should check that both use the same version otherwise shit happens (aka race cond on the dlopen of the native part of protobuf IIRC this is a limitation of the python native architecture since it loads everything in the same process using dlopen() ;))

*2) by default we compile glop as object library and integrate it in libortools so or-tools "embed" GLOP. it is just for SCIP developpers IIRC they wanted to integrate GLOP inside scip so if you disable the build of ortools (BUILD_CXX=OFF) then you can enable the build of libglop only but it is not something you should do it is a very SCIP internal workflow feature ;).

*4) 1.17.5 add a "regression" making Clp unusable in or-tools it is fixed on stable/1.17 branch which is post 1.17.6 so yes ortools-clp is broken in 1.17.5 AND 1.17.6 and hopefully should work with the "incoming" CLP release/1.17.7
ref of the issue: coin-or/Clp#130

[BastianZim]

*2) Ok then I will leave that untouched.
*4) Ok thanks for the clarification, then we'll exclude those two.

[wolfv]

@Mizux the issue you described with protobuf, or-tools and tensorflow is exactly why we have the global pinnings :)
I think it would be worth a shot to try to build with the current cf-supported protobuf, what do you think @BastianZim? Especially since we also have tensorflow 2.6 on conda-forge :)

[BastianZim]

Definitely @wolfv! I mean, that's why we have it. ;)

[BastianZim]

@Mizux

IIRC sat Python wrapper is basically based on creating a proto file then passing it to the C++ layer so if python sat samples pass you should be green ;)

Do you know if there are some in the or-tools repo or should I write one myself?

Edit: Found many in https://github.com/google/or-tools/tree/stable/examples/python Is there a recommended one?

[Mizux]

prefer to use ortools/sat/samples/*.py there are smaller and we tried to cover all basic use cases and are all part of our CI.
src: https://github.com/google/or-tools/tree/stable/ortools/sat/samples
https://github.com/google/or-tools/blob/86d4c543f717fa8716a9d66c25143660668bf825/ortools/sat/samples/CMakeLists.txt#L12-L17

On the contrary, "examples/*" is more a trash folder with lots of advanced samples (means longer run time and complex problems using few features at a time), also few of them rely on input files etc so we don't/can't run them all in our public CI so sometime they can be broken for months without notices :(

Basically all ortools/<components>/samples/*.py are good candidates to quickly tests the build of each components. (we(I) also try to have the sample available in all supported languages (C++, .Net, Python and Java) and there are the ones you can aqlso found in the guide)
e.g.: https://developers.google.com/optimization/cp/cp_example

[wolfv]

@BastianZim ready for merging?!

[BastianZim]

@wolfv Don't we first need the pinnings?

[wolfv]

I think it's fine to have the pinnings afterwards.

[BastianZim]

🎉

Thanks everyone for your help!

[lperron]

Thanks for everything.

[cdeil]

This is fantastic! Thank you!