Node.js - subdomain per user + allowing custom domain

Would this be the best way to approach this problem? Is there an alternative way?

Are there going to be any potential issues with scalability?

What you described with the wildcard DNS is the best way and you won't encounter any scalability issues caused by that. The alternative is to use a DNS host that has an API, such as Route 53, and dynamically add a subdomain to it for each user. That has several disadvantages over the wildcard though. It could become a pain to maintain, it could take a while for new DNS records to propagate, and you may quickly hit limits (On Route 53 you need to request a limit increase to go above 500 hosted zones per AWS account).

What is the best way to allow users to use a custom domain for their area within an application?

The same way you have logic to check the subdomain (checking the HTTP Host header). When a request comes in you can check the Host header and determine which user the domain belongs to (whether it is a subdomain or a full custom domain); as long as they've told you what their domain(s) are. Adding the CNAME is their responsibility (though you can give them guidance, obviously).

If you support HTTPS, you will only be able to do that on devices that support SNI. You can get a wildcard cert for the subdomains, but you will either need the domain owners to upload a PEM file or get their permission and use something like Lets Encrypt's api with the webroot domain verification check.