Lacking v3 APIs

[Suyash08Oct]

Hello, As a consumer of this cloud foundry client, I noticed that for v3 some of the APIs are lacking, which includes some of the major functional apis for operations related to apps, service instance, service plans, service keys, service brokers,space routes,security groups etc.

I would like to contribute here, to bridge these gaps from cf v2 to v3. Please help me with the procedures for the same. and please acknowledge if these changes are already in pipeline to get released.

And please share the release process and the estimated time taken for the PR approval, review and release. So that as a consumer, I can also plan to consume all the v3 related APIs in my application.

Attachment: captured some of the gaps from v2 to v3

[dmikusa]

First, let me say thanks for taking the time to put this list together. Some of these gaps were known, some were not.

Second #1157 which was just merged, should I believe complete a few of these (service instances & plans I believe). I think #1158 may cover some as well, but I haven't look in depth at that PR yet.

As far as contributing PRs, we'd love to have them.

1. You have to sign the CLA before we can accept any PRs. The first PR you submit will prompt you. If you work for a large company you may already be covered. The bot will walk you through the process.
2. Any PR to add new APIs should contain the client, reactor client implementation, reactor client unit tests, and integration tests to validate the work. The integration tests part can be a little tricky as you need admin access to a CF environment to run them. This link has some pointers for running the tests and getting an environment. I have also offered to run integration tests for folks, as we have labs to run them. I would just ask that your tests compile OK first & you are reasonably confident they'll work if you want to go this route. That'll reduce the roundtrips it takes to get things ready to merge. Implement service instances and plans creation and update #1157 is a good example.
3. As far as turn-around time. We try to take an initial look at the PR within a week of being submitted. The time to merge will vary depending on if any changes or corrections are required. Times can vary as the small team at VMware that manages this project is also responsible for a number other projects and from time to time we get delayed by work on other projects. We do try to be upfront in the PR about this when it happens. We appreciate your contributions and want to respect your time as we know it's valuable.

Thanks & let me know if you have any other questions.

[Suyash08Oct]

Thank you so much for the quick and detailed reply. Regarding PR Release, I can understand the constraints. I was following up on the time line for the release because we are currently consuming this client with the v2 version. We are planning to shutdown the use of v2 APIs . So we need to perform migration from v2 to v3 as early as possible. So if will get the rough estimates for the complete process (from PR till release) required for the release for our PRs, it will be easy for us to plan our consumption accordingly.

[dmikusa]

We are planning to shutdown the use of v2 APIs

Could you expand on this? Is that an internal choice? While CAPI v2 is deprecated, I've heard no plans or timeline around the actual removal of the v2 API.

So if will get the rough estimates for the complete process (from PR till release) required for the release for our PRs, it will be easy for us to plan our consumption accordingly.

Our present coverage, as you've seen, is not complete for the v3 API however we do have quite a few of the APIs working. If you need one of the missing APIs, feel free to submit a PR & we can typically get that reviewed and merged within a few weeks. Releases may take a little longer, but it really just depends on what is happening at the moment. We like to batch up multiple changes in a release where possible. At any rate, I would just encourage you to mention/ask at the time on the issue/PR you open and we can figure out a reasonable course of action for a release.

Where we are more lacking is in the operations library, not sure if you're using that library, but the operations library provides more high-level concepts like cf push, and most of the v2 operations library is still using v2 APIs. This still works because the v2 API is still present and operating, but there is some new functionality that's missing, like rolling deployments & rollbacks. You can technically do that with the v3 client API, but it's more work.

Hope that's helpful. Let me know if you have any additional questions.

[stephanme]

Some background info regarding the mentioned "shutdown the use of v2 APIs":

As you said, there are no plans to remove the v2 API implementation from CAPI.

However, we (SAP) plan to reduce (drastically) the CF API v2 usage on our Cloud Foundry foundations. The main reason is that the v2 API doesn't perform that well on large foundations (think of > 70T used app memory, > 150 api VMs). We fear that we hit CF API scalability limits on further growth. Our metrics show that an avg v2 request takes more than twice the time of an avg v3 request. And all investments in CAPI performance and scalability go into v3 as v2 is deprecated.

We won't switch off v2 completely in our foundations but we will add an additional v2 rate limiting and decrease the limit over time: cloudfoundry/cloud_controller_ng#2882

[dmikusa]

@stephanme Thanks. That makes perfect sense.

[suyash-sap]

Hello @dmikusa,

To initiate the process for contributing the lacking v3 API, I have forked the source code. And was going through it in detail. I have gone through https://github.com/cloudfoundry/cf-java-client#readme for the details. Is there any architectural document, which I can follow for the deep insights of the source code.

Thanks, Suyash

[dmikusa]

That's really it for documentation.

At a high level, you have the client which is generic, you have an implementation of the client which is written using Reactor, and the operations module which is higher-level operations built on top of the client, these are things that take more than one call to do, like cf push.

[suyash-sap]

Thanks for the explanation.

In client, written with Reactor, there is a fine segregation between v2 and v3 implementations.
But with cloudfoundry-operations, I am unable to find the same.

For example, If i need to do cf create service test then the call will eventually come under DefaultServices.java which is calling the v2 specific client implementation.

I need to understand the architectural design which we need to follow here, so that we can make the operations ready for consuming both v2 and v3 specific clients.

Please feel free to correct my understanding, If I got it wrong.

Thanks, Suyash

[dmikusa]

In client, written with Reactor, there is a fine segregation between v2 and v3 implementations.
But with cloudfoundry-operations, I am unable to find the same.

Correct, there is no separation of the code in operations. I think you could perhaps say that there doesn't need to be a difference, at least not per API version. The operations library provides high-level operations, which take multiple client calls. If you're using v2 or v3 or a mix of v2 and v3, it doesn't really matter to the consumer of the operations library. They just want the operation to work.

Where that gets tricky is when you can't do something in the same way with v2 API as you can with the v3 API. The cf push operation comes to mind here. I'm not totally sure that you could swap out v2 for v3 API and not have a breaking change. If there are differences in behaviors, we'd need to account for that somehow, probably with a new method or operation.

I'm not opposed to doing it differently though if folks think that it would be better to have it versioned. That would require a new structure in the code to differentiate things by API version like you have with the client. That doesn't exist in operations because I don't think any of the operations have started to use v3 yet.

Hope that helps!

[suyash-sap]

Both of the Approaches which is 1. Using v3 API directly from operations and 2. Creating a separate layer for v3 has their pros and cons, which I feel should be discussed before:

For Approach 1: Using v3 API directly from operations
Pros:

• Ease in development as it will have minor changes in the current cloudfoundry-operations.
• Ease in consuming the cf-java-client for existing consumers, as method will be same, the change will be internal to operations for v3 consumption.

Cons:

• The current design of the client has a fine segregation between v2 and v3, So changing the operations using v3 will not be as per the current architecture and it will limit the v2 to the rector client specific implementation only.
• Is the partial implementation of cloudfoundry-operations will work and can get released? where some of the methods will use v2 and some will refer to v3 for their implementations.

For Approach 2: Creating a separate layer for v3 in cloudfoundry-operations
Pros:

• Aligned Implementation with the current architecture and has a fine layer segregation.
• Consumers has an option to switch to the specific implementations.

Cons:

• Code and Efforts will get Duplicated.
• With the obsolescence of v2, the code for v2 will not get used at all.

This is what I can think of now, we can have a brainstorm here for a decision so that, we can the start the contributions as early as possible.

Thanks, Suyash

[dmikusa]

I don't really have a preference for approach 1 or 2. Each has some pros and cons, I'm not sure there's a clear "better" option.

Given that there is a difference in the level of effort for the implementation, I would leave the choice up to the individual that wants to implement and contribute the code change since it's their time and effort. In short, it's up to you @suyash-sap I'm Ok with either approach.

[suyash-sap]

With Approach 1, Is the partial implementation of cloudfoundry-operations will work and can get released? where some of the methods will use v2 and some will refer to v3 for their implementations.

[dmikusa]

Yes, I think that would be an advantage of this approach. We can slowly upgrade the library and users will just get the benefit of switching to v3 without requiring any code changes. It just emphasizes the importance of making the upgrades in a non-breaking way. If we can't do that for a particular operation, then we'll have to version that particular operation. That's still a bit less work (for us and for users migrating) though than versioning the entire operations library.