Kubernetes: What's the idiomatic way in K8s to setup a custom proxy between ingress and its services?
So after a lot of reading and googling I found that the solution was to use API Gateways that are available as libraries (preferrably based on .Net):
Ocelot placed behind the nginx ingress fits the bill perfectly
Ocelot is a .NET API Gateway. This project is aimed at people using .NET running a micro services / service oriented architecture that need a unified point of entry into their system. However it will work with anything that speaks HTTP and run on any platform that ASP.NET Core supports.
Ocelot is currently used by Microsoft and Tencent.
The custom middleware and header/query/claims transformation solves my problem. Here are some worthy links
Microsoft Docs: Implement API Gateways with Ocelot
Features
A quick list of Ocelot's capabilities for more information see the documentation.
Routing
Request Aggregation
Service Discovery with Consul & Eureka
Service Fabric
Kubernetes
WebSockets
Authentication
Authorisation
Rate Limiting
Caching
Retry policies / QoS
Load Balancing
Logging / Tracing / Correlation
Headers / Query String / Claims Transformation
Custom Middleware / Delegating Handlers
Configuration / Administration REST API
Platform / Cloud Agnostic