Unit-Tests in djinni-support-lib can not be executed without a generator release.

[jothepro]

In #43 a big new feature is introduced: Python support for djinni.

Because djinni has been split into components, we do now face a problem while introducing the big new feature that includes big changes in both generator & support-lib:

The changes in the generator need to be released, otherwise it cannot be pulled as dependency in the support-lib to execute the unit tests. Once the generator is released, the unit-tests in the support-lib repository can test if the generated code compiles & works.
If theses test fails because of an error in the generator implementation, a hotfix needs to be made in the generator and the fix has to be released again.

From my point of view this should not be a problem in general, as all things should be tested locally before being pushed to the generator/support-lib repository anyways.The tests in github action should just be there to verify that everything is fine.

But in reality things go wrong, and situations like this may get nasty, if multiple patches need to be made in the generator for some unexpected reason. As a maintainer I think we should discuss options on how to reduce this hard dependency when introducing big features that require changes in both components.

[a4z]

To start the discussion, here something, maybe we can iterate on it

1. a merge into master generate a djinni binary, djinni-latest , and this is used
   since this is tied to a commit, and the commit sha could be reflected in the version, there is some idea what this includes, if this info is ever needed.
   And having a latest, rolling release, available is common practice. Use at your own risk.

2. PRs for support lib has some way to know which generator to use. Which way this is, I do not know at the moment ...

[jothepro]

Theoretically you can always download a canary build from the latest github actions run:

https://github.com/cross-language-cpp/djinni-generator/actions/runs/760704611

Each "CI" workflow creates an artifact with the djinni binary.

But it may be hard to get the artifact by using the api, afaik there is no endpoint for the "latest" action/artifact.

[a4z]

I think we should move that into the discussion part, since it does not seem there will be any action in reasonable time.
Also, the problem is not really a problem that stops something, it is more an inconvenience, that delays the pipeline if we run into the situation that there is a dependency.

I think what we actually would need might be something like that
https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html
and/or that
https://docs.gitlab.com/ee/ci/yaml/#cross-project-artifact-downloads-with-needs

not sure if gh action provides something similar

[jothepro]

I'm really struggling to find a solution that solves our problem, apart from moving back to the concept of a mono-repo, at least for the generator and the support-lib... 😞

The core problem I cannot solve is that if I my feature requires changes in both components, both repositories need to reference an arbitrary version of it's counterpart, even from a fork. With this version they would then be able to execute all tests given the exact counterpart that they need.
But this reference would need to be hardcoded somehow, which means that it would have to be merged into main in at least one of the components once the feature is approved. This leads to a reference to somebody's fork in main, which is not desirable IMO.

Even if github would support sth comparable to a multi-project pipeline, I cannot imagine how this could work well together with forks...

One could create an uber-repo that references both components and executes tests, but at this point i think it would be easier to put both components into one repo again. :(

---

To understand how I came to this conclusion, you may want to read the following proposal for solving the problem. I wrote down my idea while thinking it through, with the result that I understood why it doesn't work:

Proposal: Add the support-lib as submodule to the generator and provide a makefile or sth similar that executes both the tests from the generator and the support-lib tests with the current generator binary in ./target/bin.

Use cases that this solves:

• I've made a change in the generator and want to ensure that it doesn't break the support-lib tests. All I need to do is run make test in the generator repo and automatically the binary is built and tested against the local static tests but also the support-lib runtime tests. The pipeline can do the same.
• I'm doing sth that requires changes in both the generator and support-lib in order to work.
  1. First I make the generator changes and write some static output-tests for it to make sure the expected output is generated.
  2. Then I update the support lib from inside the submodule
  3. I push both changes to my forks and because the submodule points to my support-lib fork, the github pipeline in the generator can perform the same tests that I executed locally and thereby confirm that the tests are all green.
  4. I can confidently merge the changes into main and do a release... The submodule still points to some arbitrary commit from my fork... Im screwed... I would have to merge the support-lib changes first, but this would not be possible because the tests would fail without the correct version of the generator. I could reference the correct version of the generator in the support-lib somehow, which would just move the problem to the generator again. I'm locked.

---

Sry for this wall of text. Hope it helps to understand my problem. Maybe there is sth that I am missing? 🙏

[a4z]

I understand your problem since a lot of people struggle with such situation, and then do the wrong thing, like creating monoliths.
And have a lot of effort with other thing after that decision.

There is no technical problem, just a time problem that prevents me from implementing on of the multiple ways of solving this situation.

Let me go one thinking path: gh actions should be able to access an artifact from a different repo,
can we get the latest artifact from the most recent successful main branch built?
In this case, as soon as a PR on generator is merged into master, the PR for the support lib will be able to succeed by getting the generator artifact from the newest successful master without waiting for a release.
(optimal , for the PR of support lib, it would be good to have a toggle if the latest master or newst release is wanted)
If gh action can not do that, then we do something else, but this is not a reason to make component layout decisions that are simply not good.

When I have more time, I can bring some other ways of getting up a infra that potentially solve the problem in one or an other way, but today it's too late and it might be too verbose to write anyway. We can, and should have a voice chat about that topic.

[a4z]

Shorter, more concrete version:

All generator versions that a support lib would need have already been built and exist as artifact.

Use case:

• User A develops for feature X in support lib and need generator_next
• User B develops patch in support lib and need generator_stable

Development happens in parallel.

Both, generator_next and generator_stable are available as aritfacts , example for some feature build
https://github.com/cross-language-cpp/djinni-generator/actions/runs/1418947232
limitation: feature X build is only available until it expires , 90 days, but that should be sufficient in most cases, if not there are solutions
The point is, the generator is there and exists, no need to build it. But how to get it?

The question is

• How do developer A and B specify the generator their support lib branch requires and how to get that for the CI build
• What happens in main branch after merging into main
• How to protect release builds so the check happens with a generator release

I do not see subrepos add any value or helping anything
But maybe something like this can help:

https://github.com/marketplace/actions/download-workflow-artifact

[a4z]

this is also interesting: https://dev.to/oysterd3/how-to-release-built-artifacts-from-one-to-another-repo-on-github-3oo5

[jothepro]

Today I had an idea for a very radical solution of the problem.

Disclaimer: Pls don't take this idea too serious in it's current state. I had fun implementing a proof of concept to check if it could work. While I am quite happy about the outcome, this doesn't mean that I'm fully convinced yet myself. It may be a dumb idea but I'd like to throw it on the table to fuel this discussion!

Ok, let's go. 👏

The idea: If there would be only one component (the generator) instead of two, there would be no interlocking problem between interdependent components.

So how do we get rid of the concept of a support-lib component? My proposal is to not distribute the support-lib as CMake target, conan dependency or sth. else, but as sources within the generator.

I realized that the majority of the logic in the support-lib CMakeLists.txt and in the conan configuration is all about which sources should be distributed for what target-language and where they should be installed.
The same can easily be achieved with the generator: If the generator does generate output for, let's say JNI, it could as well output the required support-lib sources alongside with the generated gluecode.

I've implemented a proof-of-concept of this idea here: https://github.com/jothepro/djinni-generator/tree/merge_support_lib_into_generator

Without going too much into the details, this is how it works:

• The support-lib sources are located in the resources folder of the generator and thereby get distributed with the djinni binary.
• If gluecode for a certain language is created, the generator checks if support-lib sources are required for the gluecode to build, and copies the required sources into a djinni subdirectory alongside the generated code.
  That way the #includes in the generated gluecode still work.
• The generator-binary is located in target/bin once successfully built. Since we know where the binary is, the path can easily be fed into the support-lib integration tests.
• If for some reason this concept doesn't work for you, you can disable it by setting --copy-support-sources false and add the support-lib in another way.

Pros:

• I actually wasn't the first person steering in this direction. I realized that the way how json+extension.hpp is integrated into the generator is exactly the same concept.
• We only need to distribute one component: the generator.
  • This makes the support-sources more build-system independent, because the logic from the CMakeLists.txt is now implemented in the generator.
  • The release process get's lighter, no messing around with conan-center-index any more! 👏
  • For me the support-lib build configuration for C++/CLI never worked anyways. Currently the only way to make djinni work with C++/CLI for .NET Core is by copying the support-lib sources. This is not something that can be fixed by patching the build configuration of the support-lib, it's caused by Microsofts lazy implementation of C++/CLI in .NET Core (See here how I currently work around this)
• Development on the generator & support-lib will get easier because the test-suite of the two components are re-united and can easily be executed by the developer, no matter if he made changes in the generator or support-lib. The main concern about a fancy interaction between pipelines like @a4z proposes it is that it cannot be easily reproduced on the local machine. Once you have to touch both the generator & the support-lib, the local development-process gets quite annoying right now IMO.

Cons:

• I'm sure that this will break someones setup completely. I don't know how exactly, but I have a feeling about it...
• We'd loose all of the git history of the support-lib (again). That sucks. Maybe there is a way to keep or merge the git history when moving the files? 🙏
• Build-times may be worse, but for those 5-ish small support sources I honestly don't care.
• By removing the CMakeLists.txt from the build, we loose some information, e.g. that on macOS ARC needs to be enabled and the Foundation-framework has to be linked. But that's stuff that most likely has to be configured in the code linking the support-lib as well anyways, so personally I don't care either. And the C++/CLI configuration only works for .NET Framework and will always be useless for .NET Core (which seems to be the future of .NET).

Phew that was a lot of text. Things happen when I'm in flow 🙈 .
@a4z @freitass as we may have a meeting about this topic in the near future anyways, don't feel rushed to give me a full feedback to this literal wall of text here in the discussion. Take your time. I don't plan to rush this topic and I'll not be mad if I invested my time into a dead end. 😊

[freitass]

It took be a while to get used to the separate repo idea when I first learnt about xlcpp and I still have mixed feelings about it. I'm not completely against going back to a mono repo (maybe not for everything) but I just don't like the idea of going back and forth.

With that said, this gives us a lot of material to think and talk about. I have a feeling that there might be a simpler solution in the mono repo direction.

I'll just address the points you enumerated as "cons":

I'm sure that this will break someones setup completely. I don't know how exactly, but I have a feeling about it...

With proper upgrade instructions this is a solvable problem.

We'd loose all of the git history of the support-lib (again). That sucks. Maybe there is a way to keep or merge the git history when moving the files? 🙏

We wouldn't. There are ways to join two completely independent source trees in Git.

Build-times may be worse, but for those 5-ish small support sources I honestly don't care.

Meh.

By removing the CMakeLists.txt from the build, we loose some information, e.g. that on macOS ARC needs to be enabled and the Foundation-framework has to be linked. But that's stuff that most likely has to be configured in the code linking the support-lib as well anyways, so personally I don't care either. And the C++/CLI configuration only works for .NET Framework and will always be useless for .NET Core (which seems to be the future of .NET).

I'll comment on this later because I don't think I completely understood it.

[a4z]

with local development there is no problem at all

PATH=../generator/assembly/bin cmake ....
or you pass ../generator/assembly/bin/generator to the cmake of the support lib so find_program picks that up
I hope we agree that there is absolute no issue to use whatever branch/build of the generator you want to use for local development.

with CI, I explained, all artifacts are there already ready to consume
It's just get the one you want into your CI build.
That's all, and I am surprised about the complexity of suggested 'solution' instead just solving that clear focused simple issue.

Let's re-write the world because this CI does not give me out of the box what is wanted?

About putting the code into the generator, just one example since I am limited in time: I do small development in the support lib and need to go through a scala generator build for every small change? No thanks .
And there are more issues with that approach, I just have not the time to write them all down, that's why I suggested a discussion about that.

My suggestion is: State the problem right.
The problem is, we have currently no way to get an existing binary from the artifacts in the support lib build if something else is needed than the latest release.
So let us solve that issue, before re writing the world and mess up things that are actually pretty fine.

[a4z]

Just would like to add, I think basically your idea is good, @jothepro , but we evaluate it after solving the current problem in an extra way.
Not because of the CI build (non)problem, but in the context of choosing implementations, like the json parser, or other interesting options that could be solved in that way.

[jothepro]

with local development there is no problem at all

Sure, there is no problem. It just doesn't feel straight forward to me and there may be room to improve the developer experience.

I do small development in the support lib and need to go through a scala generator build for every small change?

No you don't have to do that. I moved the CMakeLists.txt from the djinni-support-lib with the sourcecode, so the C++ development experience should be the same as before. Only the test execution currently has to be triggered by the Makefile that I improvised in the root directory. Being able to fast and easily compile and test the support-lib for development is a priority, I agree!

My suggestion is: State the problem right.

I'll try to state my POV of the problem. (Sry for having to bother you with a wall of text for that again. I've tried to give a complete picture of my POV to find out if we all have the same foundation for the discussion! 🙈)

In the original Dropbox repo the support-lib sources where mixed with the generator. In order to use the support-lib you had to clone the repo.
We identified that generator & support-lib could be treated as separate artifacts with separate release cycles & repositories so they could be served as prebuilt artifacts via conan & friends. This makes it easier to integrate those artifacts into a build system without having to worry about the internals, which is a great achievement IMO! 👍

But now both of these components are required in the correct version to be able to work well together.
Currently we are only able to test the entire system in the support-lib repository, because that's where the (real) tests are located and where we specify what generator version fits the current support-lib version. The limitations of this approach are the reason for this discussion.

The original solution discussed here is to pull in an arbitrary version of the generator into the support-lib test-CI, to be able to execute the tests there and thereby being able to verify changes in both the generator & the support-lib. That's really appealing and straightforward at a first glance.

The reason why I propose another (admittedly radical) solution to the problem is that I think that our current situation is only the symptom or a different underlying misconception: There currently are separate artifacts (generator & support-lib), but they cannot work without each other and are highly interdependent, which makes it nearly impossible to properly test them isolated from each other. This problem only became more apparent to me with the Python gluecode that we now need to distribute as an artifact somehow (cross-language-cpp/djinni-support-lib#57):
I absolutely agree that we need to separate the Python code into it's own artifact & repository if we follow our current architecture. But how and where do we test the code?
In the pip repo, pulling in the support-lib & generator?
In the support-lib, pulling in the pip & the generator?
By building up a submodule hell where all components get pulled in from source?

IMO the consequence of following this path is an increasingly inflexible system that makes it hard and labor intensive to introduce bigger changes and that is hard to grasp for new developers.

My proposed idea to eliminate all code artifacts and merge them into the generator is the consequence of me realizing that on one hand the ties between the generater & support-lib artifacts are too strong to call them independent components, while on the other hand I'd like to avoid going back to the initial monorepo situation where the sources of multiple components are mixed up and the resulting artifacts would have to share the same release cycle even if no change was introduced in one of them.

And there are more issues with that approach

While my idea looks intriguing to me at a first glance, I am, like you, skeptical if what I propose is really a viable solution that will work well outside of a small test setup. I'd like to find and evaluate the risks and problems that come with it, so whatever problem comes to your mind, pls let me know about it! 💪

[a4z]

it shall be possible to make a generator release without a support lib release
it shall be possible to make update to the support lib without having to update a genarator
a change to support lib should not trigger a build of generator if its not required
a change to a generator does not require a rebuild a the support lib if its not required
both should be able to be released with out releasing the other
(now add pip to the game and repeat the same with 3)

you can keep things extra/separated and pack them easy into a mono thing
you keep things together in 1 repo, it's very hard to handle them extra/separated.

I know very well about mono repos and their pros and cons,
Please remember, it is me working on the bazel port, (which will result in a mono repo similar thing at the end)

let me repeat what I have written else where

local development works just fine, there is no problem

we are talking about CI stuff that is easy to model, if we really need
(currently we encounter that problem so seldom that I think we have lots of other things with higher prio to solve)

A PR of a support lib might either want

• A tagged generator build
• The latest build from generator main
• Some PR build (not yet in main)

A Main branch from support lib might either want
(Main branch == release branch, there could be more, e.g, next feature version + current patched version )

• A tagged generator build
• The latest build from generator main

A tagged support lib wants

• A tagged generator build that fits

I think it should be easy to add that info to a build and let the CI select the wanted option,
Then builds will not be require to wait.
I just see no priority in investing time into it since I think the other work, like the python/pip and bazel build, is more important.
I also can think about (see) a bazel build that let me compose such things simply, and then all variations exist, this is why I first want to have time for the bazel build(s), since they might give us already some solutions (bazel works nicely with external repos)

Generally speaking, this is the second time in more than a year we have a PR where we want to test the generator changes in the support lib.
The second time of how many PRs?
Usually it is not a good idea generalize the edge case to the common case, a rule for software design, but also infra design.
This is why I think we currently make a mosquito look like an elephant by focusing on that edge case and I would prefer to continue work on my current djinni tasks

[paulocoutinhox]

As contributor of both projects (cross and snapchat) i want leave my opinion too.

I agree with join the projects into a single repository (generator + support lib).

I see this in other projects, like Snapchat (https://github.com/Snapchat/djinni) and it works very well.

The release of both things can be in sync if they are together and it is easy to know what work in the current version.

I don't know how many users use the conan version of support files. I use djinni and c++ on mobile, desktop and wasm in my job and never use that repository(mainly because it's the only repository that doesn't work well with darwin-toolchain, but this is another problem). I always have a copy of support-lib in all projects to don't depend of external things. To me this can be removed from conan center index, because don't make any difference (in my opinion).

In my opinion, separate the projects make our life hard do keep the generator working and well testes with support-lib.

For example: In a real example, im trying to add WASM support, but i can't use the support-lib to test it, i can't have examples locally to test it and other things because the djinni parts don't talk with others (generator + support-lib).

Sometimes we want to improve or organize something and we end up giving up simplicity and efficiency.

I totally agree to keep things together.

And if really need keep it separated, keep a copy separated when is stable in main repository only to work with conan cci. The repository with a copy can be a more detailed examples etc to test it and the main repository with all together make things only to check if it is working, without a lot of examples.

[a4z]

For the rare cases where we have development where generator needs support lib, and vice versa.

• Fork generator / create branch feature/wasm
• Fork support lib / create branch feature/wasm
  When running the cmake for the support lib, you can make cmake find the output from the generator fork
  -> develop
  you can also make a parent cmake and add support lib as a sub project with cmake to build your sample app (and find the generator the usual way. -> develop

So for local development there is no problem. I hope we agree on that.

Now, are you complaining about the limitations in github actions, that has no multi project pipleines out of the box as other services have, for example https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html
and suggest therefore putting apples and bananas into one box ?

The compare with snapchat is not suitable, this is monorepo per default, if you suggest to throw away sbt, cmake and explain me how to nevertheless make the project a general purpose solution (what snapchat is not, but this one is), and accessible to all users, I am willing to go that path.

Please do not forget, we have much more possible use cases than snapchat , who as exactly 1, their company software
For example:
As a user who get a project with generated source files to implement the details.
I only want to download the support lib, build it for my target, have as fast build times as possible
Generator is not my concern since generation has been done by an other project.

There is for sure a way to satisfy also that via a single repos, but before mashing all together, I would like to know how that works.

And please do not forget, there is also the possibility to make a new repo in the organisation, and make the monorepo as a new variation, and then see what can be archived with that approach.
But throwing everything away and running blindly into a different direction because of problems with tool usage and gh action, ....
I think it's better to first make a working draft and see how that works ... if it's better, we can change, but that should happen in parallel to the current way

[a4z]

Since my opinion on the the topic is not very clear, what it possible since I never stated it that clear, please let me state it explicit.

I do not think a monorepo as the original Dropbox repo was, or the current snapchat repo is, is a viable solution for our use case.

But I think the idea of @jothepro he described up here is a good candidate for a solution to the problem. And not just to fix the CI problem, but a general one, see my notes to the python implementation

On a first glance I had some concerns, since it introduces quite some change, and I still do not have a fully opinion on all the changes it might introduce, but maybe it's time for a djinni-NG (next generation?)

I did not expect a new language landing that triggers the use case that the support lib CI needs the latest staging build from the generator so fast, since so far this happened super rarely.

And now I wonder how to distribute the limited available developer time we have.

• Fixing the CI issue, by making PRs building with the latest master build of the generator, and everything stays at it is

or

• looking into Johannes solution, evaluating it and see how to integrate it. There is also a little bit the problem that for existing projects that could be a breaking change, but mostly likely only for the build system, by removing finding and linking a library, and that could turn out to be beneficial.

Fixing the CI has the advantage that we can estimate how much it will take to implement, and it will have no side effects to the project in general , the existing deliverables stay unchanged. Also existing developer workflows, but it does not solve some general problems.

Johannes solution has the advantages that I think it is a better solution, but since it's a huge change. I am unsure about unexpected side effects we would have to fix, and changing the deliverables. Even if I think this is an improvement. Also the mentioned change that we do not longer transport dependencies via cmake .. but in worst case, that would be a fixed via documentation.
I am also a little bit worried about development setup, for local development since I fear longer test cycles when working on the support lib code (since it must build the generator) , .also IDE setup .. , but that all can be solved very likely in a good way.
So there is a little bit of uncertainty, but I think it's the future

How to continue ? Djinni NG or to preserve the current structure and deliverables ?

Opinions are welcome!

[a4z]

2.5 years after the topic has been open, long-time usage has shown:
The rolling release of the main branch builds solves most problems when a non-released generator is required.
For the super rare use case where a pre-mereged generator would be needed, downloading a CI artifact can work sufficient. We never had this use case, but soon we might.

After years we see that this use-case is the absolute exception.
So far, nearly all of our releases have been independent of each other. This is the common case.
Generalizing the exception / edge edge case to the overall solution is an architectural mistake, and I am happy we could avoid it.

This will also be helpful when we add cmake helpers that use cmake's FetchContent to either include one, or both components.

[a4z]

For the support-lib: Get the generator I want in CI (release, pre-release, or developer snapshot)

In a Pull-Request:

Some changes might require a specific or new generator to be used in CI.

In this case, as a user, I want to be able to get the generator I want in CI.
This might be the latest release, the latest pre-release, or a generator from a specific pull request on the generator's repository.

The pull request might also contain changes to the GitHub workflow file.

Pre-assumption: The repos stay separate since :

• mono repos are a special case that does not apply to this use case
• it shall be possible to quickly come up with an extra generator implementation that re-uses the current support lib
• generating the support lib code via the generator might be a future for djinniV2, when the Swift-C++ interopt is more mature, and only a JNI part is needed. But that might then be an extra project.

Possible solution: Download a generator as release, pre-release, or from a pull request

Technically, there is no problem.

Example how it could be done.
This example has a simple Python script that downloads the generator based as specified in a generator.cfg file.
If the selected generator is not a release, a final CI step will fail, but all other CI steps will run.
Implementation details can be different, but the concept should be clear.

So, it is possible to download the generator as a release, pre-release, or from a pull request since the generator is available as a build artifact.

However, there is a limitation from GitHub:
Getting a build artifact from a pull request requires to be authenticated.
This is not possible with a PR from a forked repository because, in this case, access to the repository secrets is not available.

Possible solutions:

• Work in a branch of the generator's repository
• Use pull_request_target instead of pull_request in the GitHub workflow file

Work in a branch of the generator's repository

It needs to be evaluated, but in theory, there should be no problem.

In case a PR generator is required, we open a branch in the djinni-support-lib repository.

We can then either:

• work directly in that branch
• open thi branch as a PR and have a fork working against that branch

Use pull_request_target instead of pull_request in the github workflow file

It works, but there are problems if we want to update the pipeline file since it is always taken from the main branch.

Way forward

I am open to talking about details, but I also would like to fix this topic and close it finally.
Whatever is for an alternative implementation, or version 2, could be possible is a different topic and should be handled separately and not be projected on the status quo of the current project.

If no input comes, I will try to work in a branch of the generator's repository.

There is a time restriction since I would like to have the problem solved for the support-lib PR that made that issue actual. (webassembly support)

[paulocoutinhox]

Generate the support-lib by the generator or have a mono-repo, don't go in the same way? I understand that the both things only work better when they are together. I vote up to join it, if possible.

[a4z]

Generate the support-lib == copy files, if you do that you have per definition a mono repo

Mono repo where you still have generator and support lib as targets: I have the feeling that the reasons and the basic concepts why keeping them separate have been discussed in detail when the split was done.

The limitation of GitHub actions will not be a reason for a design decision that influences the layout of the repos. That's not a technical reason for a design decision.
Being able to have various generator implementations for a support lib, or being able to build and ship both independent so changes to the generator do not effect a done verification on a support lib (some projects have that requirement, maybe you never worked in such an environemnt) , is.

I start to get tired by always come up with the same input on the topic.
There has been a suggestion how to handle the rare problem of a dependency between generator and support lib.
Not respecting them, and exploring one of the possible paths forward, is quite unproductive.