Kestrel vs HTTP.sys - I've highlighted the fundamental differences below.

(The words are Microsoft's and I've only edited it for brevity and clarity. See the sources linked at the bottom).

Update:

Kestrel previously always required the use of a reverse proxy with edge deployments (exposed to traffic from the Internet) for security reasons. With Kestrel in ASP.Net Core 2.x this is no longer the case. Take a look at the documentation for more information. Kestrel Web Server Documentation

Weblistener was renamed HTTP.sys in ASP.NET Core 2.0

Sources:

1. Docs.Microsoft.com Web server implementations in ASP.NET Core
2. Docs.Microsoft.com HTTP.sys web server implementation in ASP.NET Core

HTTP.sys is windows-only HTTP / Web Server for ASP.NET Core that allows you to expose the server directly to the Internet without needing to use IIS. HTTP.sys is built on top of Http.Sys ( the same mature technology that also powers IIS' HTTP Listener) as is as such very feature rich and provides protection against various attacks.

Kestrel on the other hand, is a cross-platform web server for ASP.NET Core that is designed to be run behind a proxy (for example IIS or Nginx) and should not be deployed directly facing the Internet. Kestrel is relatively new and does not have a full complement of defenses against attacks. It's also not as feature rich as HTTP.sys and comes with timeout limits, size limits and concurrent user limits.

In essence, the choice comes down to your web application's Deployment scenario.

HTTP.sys Use Cases :

Kestrel Use Cases :

Following comparison will help you to choose which one is better

The ASP.NET 5 documentation (created by Microsoft on August 25, 2015) found here lists the chart found in the other answer (see page 107 of the bottom right book pages, but page 111 of the PDF): https://media.readthedocs.org/pdf/aspnet/theming/aspnet.pdf

Kestrel in general has better performance, if you used for one of the following below:

• Great option if used in conjunction with a reverse proxy for apps exposed to Internet
• Internal apps connecting with other internal apps on a private virtual network (not exposed to Internet)

WebListener is more secure, slower, and has more features. It is used in these cases:

• Expose app to the Internet but can't use IISRequire higher security and exposing server directly to Internet.
• Additional features: List item, Windows Authentication, Port sharing, HTTPS with SNI, HTTP/2 over TLS (Windows 10), Direct file transmission, Response caching