Let's wait to proceed until @dfokina returns and we have clearer guidelines on the haystack-experimental API documentation. While there's consensus on not providing general component documentation for the experimental package, we need to clarify whether to generate API documentation nonetheless. cc @julian-risch @shadeMe

@shadeMe we agreed yesterday during sprint mid-check to add generated API docs to https://docs.haystack.deepset.ai/reference.

As experimental classes will change during haystack-experimental lifetime we need to generate these docs continually and frequently for all post 2.3 releases. In other words, haystack-experimental API docs across all haystack releases will always point to the latest haystack-experimental. @dfokina we need your help in setting up this

@dfokina here is where I need your help the most in this PR. I've used a new category slug called haystack-experimental and we want it to show up in the left navigation pane below "Integrations API"

What do we need to do to make this happen?

@vblagoje I will need to manually create a category with the same name in Haystack docs, then the reference will be synced to this slug :) Could we name it experimental-api instead, for consistency? Or perhaps experiments-api is even better.

Ok, cool, you have more context and insights here - please name it as you see it to be consistent and logical

Just created the experiments-api category :)

@vblagoje left a couple of comments 🔝

And as the last step - we need that README_API_KEY set in this project and we should be gtg.

@shadeMe @dfokina wdyt you approve this PR, we integrate it to main, check the online docs, and we see how to go further from there?

Ok we now fetch versions from readme API just like in integrations and we can generate experimental docs in whatever haystack release we want. What remains is to do this only on pypi release @silvanocerza

@shadeMe I've thought a bit more about our approach given the independent release cycles of haystack and haystack-experimental. To manage compatibility effectively, we should eventually maintain a compatibility matrix within this project. Here's an example:

So that tomorrow when we push a new release of haystack-experimental in pypi we update docs potentially across more than one version of haystack docs. Say we are just about to release haystack-experimental 1.9. We need to update docs for both 2.4 and 2.5 in that case.

Am I overcomplicating this?

@vblagoje I wouldn't go this far, because versions roll out gradually in Readme, so it'll become unnecessarily complex to remember to push to other versions as they come out. I vote for only keeping the latest docs in the docs.

Ok then that would be much easier @dfokina I'll let @shadeMe make the final call here.

@shadeMe I've thought a bit more about our approach given the independent release cycles of haystack and haystack-experimental. To manage compatibility effectively, we should eventually maintain a compatibility matrix within this project. Here's an example:
haystack Version Compatible haystack-experimental Versions
2.3 1.0 - 1.4
2.4 1.8 - 1.x
2.5 1.8 - 2.x

So that tomorrow when we push a new release of haystack-experimental in pypi we update docs potentially across more than one version of haystack docs. Say we are just about to release haystack-experimental 1.9. We need to update docs for both 2.4 and 2.5 in that case.

Am I overcomplicating this?

That would indeed be overcomplicating it - The only compatibility assurance that we provide for this package is that it works with the latest public haystack-ai release. Let's just keep the latest docs on the website.

Ok, I agree - so this means we push to latest minor public release and unstable and as we roll releases forward we then remove them from the non-latest release where we previously pushed e.g. we would remove them from 2.2 once 2.3 is released.

@dfokina @shadeMe @silvanocerza note the latest CI run after the last commit. We run doc sync for the two latest release:

• latest release (i.e. 2.2 now)
• latest unstable (i.e. 2.3-unstable now)

That's what we want. Just need to hook them into pypi release job now.
Delete we don't have to solve today, @dfokina tells me it needs an update anyway

@shadeMe I suggest we integrate this PR as it is to see how docs turn out on the website. Then when we are happy we can move the docs sync push to pypi release hook

Looks good, I would just change the api_docs.yml workflow to sync all 2.x version. Feel free to merge after that change.

But wait @silvanocerza users will not have haystack-experimental installed by default in all 2.x branches (only starting with 2.3) and we don't even know if it works well (in fact it likely doesn't) with versions 2.1 and 2.0 as we intro-ed Secret API.

So in fact - the above hardcoded 2 should perhaps be a parameter now that we are mentioning it, and set to 1 - so that we only sync docs for 2.3 - i.e. only Sync docs with Readme / sync (2.3-unstable) (pull_request) should run!
cc @shadeMe @dfokina