How to limit memory size for .net core application in pod of kubernetes?

You should switch to Workstation GC for optimizing to lower memory usage. The readiness probe is not meant for checking memory

In order to properly configure the resource limits you should test your application on a single pod under heavy loads and monitor(e.g. Prometheus & Grafana) the usage. For a more in-depth details see this blog post. If you haven't deployed a monitor stack you can at least use kubectl top pods.

If you have found out the breaking points of a single pod you can add the limits to the specific pod like this example below (see Kubernetes Documentation for more examples and details)

apiVersion: v1kind: Podmetadata:  name: exmple-podspec:  containers:  - name: net-core-app    image: net-code-image    resources:      requests:        memory: 64Mi        cpu: 250m      limits:        memory: 128Mi        cpu: 500m

The readiness probe is actually meant to be used to tell when a Pod is ready in first place. I guess you thought of the liveness probe but that wouldn't be adequate usage because Kubernetes will kill the Pod when it's exceeding it's resource limit and reschedule.

Use the environment variable COMPlus_GCHeapHardLimit

Documentation https://docs.microsoft.com/en-us/dotnet/api/system.gcmemoryinfo.totalavailablememorybytes?view=net-5.0

And notice: you should use heximal values

It means the value 10000000 is 256MB!