Discussion on what microservices route to follow

[amarinso]

Currently we are reviewing a PR coming from a real engagement from Carrefour where they have chosen a microservices architecture based on the Netflix stack and running on top of OpenShift.

There are some archetypes for creating infrastructure for microservices as well as some configuration and REST client usage.

There is an obvious overlap between what Spring Cloud provides and what the deployment platform provides, like service discovery, routing or configuration... the same overlap (or similar) will exist with other options like docker Swarm.

We should discuss what will we offer from devon (or OASP)

• A complete although specific and biased solution (say spring cloud)
• A small set of utilities that could be re-used on different scenarios
• A stack tightly tied to a platform (kubernetes or swarm)

I'm more in the line of providing utilities/helpers/accelerators that can be used on different scenarios... but that may be insufficient for most of the engagements

Maybe we can propose several alternatives coming from different engagement experiences on the form of documentation (or white books that @ivanderk was proposing)

On my personal opinion we could bet on a platform and favor it on our engagements. In this case I would prefer to rely on OpenShift as the deployment platform because there is already a lot of effort in that project and you have RedHat behind it what gives credibility.

Please share your ideas here so anyone can be aware of our thoughts.

[maihacke]

Here is some input from my current engagement

Netflix OSS

• We started with a similar technology stack (spring cloud, hystrix, eureka, ribbon, ....)
• When having a more closely look at the infrastructure we decided for spring swarm, to solve deployment, scaling, ...
• Swarm brings it own solution for service discorvery. In my opinion it is better to solve this point at the infrastructure level. Swarm and Kuberentes do definitely solve it and there is now need to use eureka (-client)
• We chose Swarm in favor of kubernetes because of the very bad experience we had with kubernetes for the iCSD production line.
• We are still using ribbon for timeouts and retry.
• In general you should be aware that the whole netflix oss stack is deprecated. You will find a lot of discussion about that on github (Standardize on Spring Retry to replace retry from deprecated ribbon client. spring-cloud/spring-cloud-netflix#1295, https://github.com/Netflix/ribbon#project-status-on-maintenance). I think you should be aware of that when starting new engagements (or selecting a proposed solution for devon). We are currently investigating if it is possible to drop ribbon instead of spring-retry.
• We had a look at feign to. It offers some cool features, but it is very proprietary. I'm very sceptical if it is future-proof, looking at the discussions above. So we choose RestTemplate. It is not as comfortable as feign, but is very good integrated into spring cloud, too.

Openshift vs swarm vs kubernetes vs ...

• Regarding openshift. I know at least one engagement where it is used in Germany, and they have major problems with it (so we should be careful). We should check what benefits it gives us on top on swarm or kubernetes and we should incorporate the experience we have with the production line.

[amarinso]

Can we know a bit more about what kind of problems is that OpenShift engagement facing?
Also some detail about what made kubernetes a failure for the PL... Kubernetes is maybe too low level thus my preference for OpenShift (two big customers here, Carrefour and Inditex are adopting it... and we know also that Santander Bank is using it since long time ago....)

As another alternative, checkout this article where I discovered about envoy
http://blog.christianposta.com/microservices/microservices-2-0/

Also the REST client of choice may be questioned here as currently the default choice should be CXF as proposed per OASP.

[lferna]

Hi,

In our case (same as @amarinso Engagement) we could give some feedback about our (short) experience:

• We're using RestTemplate instead of Feign. I think RestTemplate has more flexibility rather than Feign, but I think Feign is more comfortable. In our case, we are looking a solution to make RestTemplate the most generic we can to re-use in others MS.
• We're working (like Devonfw) with JWT authentication and I think JWT has adapted well for MS security. JWT has many solutions for any libraries.
• We're using Hystrix to implement Circuit Breaker between MS. We're looking a possible solution to implement Hystrix in Zuul module, for externals calls. There are many opinions about this topic.
• We have integrated Swagger to document the rest methods. Our API is going to be consumed by many clients.
• We have our MS deployed in Openshift (manually at the moment). We are also looking for the way to automate it with Jenkins, Docker.

And something more that I possibly forgot.

[hohwille]

In my project we are using yet another approach. We are using Apache CXF also for the client. Also we have modularized the projects and have an api module for each microservice. We do code first (current OASP4J documentation) instead of contract first (swagger) but that should not really matter. If one microservice/app depends on another, it imports its api. Then we have a component (as a simple interface) that can instantiate the RestService from its interface via a single method defined in the components API. This internally solves service discovery, HTTP header control (e.g. correlation id), security/authentication (in our case passing JWT), error handling and exception mapping, etc. Also mocking in tests (JUnit) was addressed and solved in our project.
There are a lot of problems that need to be solved here and some might not even yet be aware of them. My proposal would therefore be:

• Add an empty marker interface RestService to oasp4j-rest.
• Add a spring boot autostarter module to oasp4j that automatically configures CXF according to our conventions and registers all spring beans that implement RestService interface - replacement for https://github.com/oasp/oasp4j/blob/develop/samples/core/src/main/java/io/oasp/gastronomy/restaurant/general/service/impl/config/ServiceConfig.java
• Add an interface RestServiceClientFactory to oasp4j-rest with this method:

<T extends RestService> T createClient(Class<T> serviceInterface);

For async support IMHO also

<T extends RestService> T createClientAsync(Class<T> serviceInterface, Consumer<?> consumer);

• Add an interface RestServiceDiscovery (suggestions for better name welcome) to oasp4j-rest with this method:

String discoverUrl(Class<? extends RestService> serviceInterface);

or more flexible:

DiscoveryInformation discover(Class<? extends RestService> serviceInterface);

• Add an implementation of RestServiceClientFactory that can be used by default and be extended as needed project needs. It will intern use the injected instance of RestServiceDiscovery. IMHO also the "security/HTTP Header tweaking" stuff should be flexible, expendable and replaceable via some component interface. By default correlation id has to be supported and a solution for passing security token (JWT, OAuth, etc.) should already be contained (via API from opas4j-security).
• Add a new OASP module for netflix stack that provides implementation of RestServiceDiscovery using eureka.
• Implementations for other stacks are welcome by contributors.

WDYT?
BTW: To already answer the question "are you proposing to create a framework by OASP?":
Yes and no. This is optional stuff and the API (empty marker interface) is so minimal that it IMHO is no framework but the value of doing so is quite high. Without such thing we have many problems unsolved in OASP (e.g. if you extract interfaces from REST-Services then OASP will currently fail because it detects services for registration via JAX-RS annotations in implementation that will then however be in interface, etc.). An empty marker interface does not interfere but solves this elegant. Projects who do not want to couple this way can still copy the entire solution into their own project and have the interface in their own project namespace (that is what is currently happening but preventing reuse and causing problems).

[amarinso]

Several comments:

• We should decide if we favor contract-first or code-first for microservices. IMO one strong point for microservices is the decoupling of the applications dependencies... and introducing the API as code may cause project to be dependent... of course you can always copy&paste

• Do we realy need the marker interface for endpoint registration? Is the code in ´ServiceConfig.java´ on @Bean public Endpoint still necessary? Because starting an application from the archetype and not having this method... automatically registers all the Rest endpoints

  • Also, could be possible to get the intended interfaces by their package name? (if following OASP4j naming)

• I'm not confortable with another layer of abstraction over client generation... If people chooses to use CXF, they will get better information and understanding without our thin layer.

• We should make aware on the documentation the concerns regarding microservices, as you mention on your comment:

  • Service discovery
  • Error handling
  • Retries
  • Load balancing
  • Correlation ID
  • Token relay
  • Testing

Reviewing CXF I came across this feature I didn't knew fail-over wich pointed me to this Apache project I also did not know about polygene and you may find interesting (Composite Oriented Programming for domain centric application development)

[maihacke]

@amarinso: Because of the problems with openshift you may ask @KarlPrott. Because of the problems with kubernetes and production line you may ask Tobias N.

Looking at the discussion here, it looks to me that every project has a very different approach. Maybe it is to early to give projects a one-size fits all solution, as we tried with the server side stack.
May it would be a good first step to give projects a guidance in the form of:

When to "do" microserivces, when not?
What topics need to be addressed including advises for "proven" technologies.
Service Discovery
Routing
Security
Logging / Tracing / Monitoring
Handling Persistence (e.g. for databases, messaging, ...)
Application design principles
...

Maybe some of the topics a covered in the microservices book...

[maihacke]

P.S. what could be helpful would be a comparison of different "PaaS" solutions, Swarm, Kubernetes, Openshift. I'm always a bit puzzled when trying to find out what exactly the differences are, and when to choose e.g. Openshift in favor of swarm. At the moment the most obvious point is that openshift offers some additional services (e.g. CI, ...), but is that all?

[hohwille]

BTW: If you still consider my proposal to lead to a framework and that should not be part of OASP for some reason then should I contribute to springframework instead of OASP or implement it in mmm?

[amarinso]

@hohwille Can you share a more elaborate example from your project to properly evaluate it? maybe I under-valuated from your comment.

We are recommending using standards where appropriate and for REST clients now JAX-RS 2 comes with this support and has some alternative implementations (CXF, Jersey, RestEasy)... what do we miss here?

Maybe we can start listing the "requirements" that we want for our clients, to add to the already mentioned:

• JAX-RS compliant
• Alternative/easier usage pattern (some frameworks provide proxy alternatives for clients and also annotation based clients like Feign/Retrofit)
• Customizable marshalers
• Cache
• Gzip support
• Customizable timeouts
• Customizable headers
• Async options
• Swagger support
• HTTP/2 support
• Android support

Some other Rest Frameworks I've come across include:

• Retofit (http://square.github.io/retrofit/)
• Restlet (https://restlet.com/open-source/)
• UniRest (http://unirest.io/java)
• OkHTTP(https://github.com/square/okhttp)
• HTTP-RPC (https://github.com/gk-brown/HTTP-RPC#java-client)

[amarinso]

Along this thoughts (calling microservices) I highly recommend this blog post from Christian Posta (RedHat)

http://blog.christianposta.com/microservices/the-hardest-part-of-microservices-calling-your-services/

[hohwille]

I already started on my oasp4j fork for a PR on this:
https://github.com/hohwille/oasp4j/
Please have a look, get involved and lets work together (not only with words but also with code).

• JAX-RS compliant - yes, server and client is JAX-RS compliant. Internal API from CXF is used as JAX-RS does not provide what we actually need here.
• Alternative/easier usage pattern - I am working on a spring factory that allows simple @Inject private MyRestService myRestService. However this approach can only work when using marker interface as I have to tell spring for which types the factory is responsbile.
• Customizable marshalers - All already solved by JAX-RS, Jackson and CXF.
• Cache - nothing done yet. However, this is always the hardes part. I am not likely to offer this as I see bigger chances to cause harm than use with this. IMHO it is always better to not mix this into the service but build that on top. Otherwise you lock yourself out if you have a demand to ensure that you get the most recent data but some cache magic is in the way and you can not pass by. However, an important issue for further discussion.
• Gzip support - matter of CXF config.
• Customizable timeouts - matter of CXF config.
• Customizable headers - already solved, see my code.
• Async options - as I proposed I will go for this once we have the basic stuff (sync calls) complete.
• Swagger support - IMHO no impact. With swagger you are generating the REST API instead of writing it by hand. Simply use the ServiceClientFactory on the REST API generated by swagger.
• HTTP/2 support - nice one. However, this is not to be done by us. We need support for this in CXF. I guess we have to wait for Java9 and Spring5 for getting started with HTTP/2. But yet, this is going to be really cool.
• Android support - I am not in an android. My experience with android is based on private play apps. We need to find some project that needs this and get into harvesting.
• Some other Rest Frameworks I've come across - I went through all of them but none of these appears to be helpful for what I assume we are aiming at. UniRest might be a nice tool for technical JUnits on services, though.

[hohwille]

Along this thoughts (calling microservices) I highly recommend this blog post from Christian Posta (RedHat)

👍

We’re implementing lower-level networking functions at the application layer

I fully agree. So why do we need netflix service discovery in client if docker swarm or kubernetes or mesos can already solve this transparently? Why do we need netflix ribbon client side loadbalancing?

[hohwille]

Angel suggested to consider side-car. Here you can find a quick intro:
http://blog.christianposta.com/microservices/00-microservices-patterns-with-envoy-proxy-series/

I very much like that you get a solution that focuses on a lot of different aspects that can be configured without the need to put all of that into your code. This makes the solution reusable across all technologies (Java, .NET, JavaScript/Node.js, etc.). However, we would need to do a verification (ideally in a real world project). Also we need to see that this solution really works cross platform easily as it is written in C++ - so if we have a java app that we can deploy to machine X will side-care also work on machine X OOTB or do I have to compile it myself and even end up with gcc errors?

However, instead of building everything into your app this could be a really nice approach to consider.

As @maihacke stated there are a lot of opportunities and so far no single best-practice. My proposal that I have started as a fork is flexible enough to work with all of these opportunities. However, I would love to also crystallize a best-practice to focus on and provide as default implementation.

[andihess]

How do you evaluate Apache Camel in that context? The Enterprise iPaaS team is using this as their approach to Microservices (with Consul as service registry). I have not looked into their details but Google lead me to articles like e.g.
https://dzone.com/articles/microservices-with-apache-camel