How does Pokèmon Go uses custom Google map using Google Map API?

I don't know how Pokémon GO does it, but you can get a really nice looking effect with some layout tricks. I'm using a sky image and I'm covering the sky image and the map with a gradient view (to simulate the horizon) and the map with a semitransparent green view.

activity_maps.xml

<LinearLayout    xmlns:android="http://schemas.android.com/apk/res/android"    xmlns:tools="http://schemas.android.com/tools"    android:layout_width="match_parent"    android:layout_height="match_parent"    android:orientation="vertical">    <RelativeLayout        android:layout_width="match_parent"        android:layout_height="170dp">        <ImageView            android:layout_width="match_parent"            android:layout_height="match_parent"            android:scaleType="centerCrop"            android:src="@drawable/sky"/>        <View            android:layout_width="match_parent"            android:layout_height="20dp"            android:layout_alignParentBottom="true"            android:background="@drawable/gradientsky"/>    </RelativeLayout>    <RelativeLayout        android:layout_width="match_parent"        android:layout_height="match_parent">        <fragment            android:id="@+id/map"            android:name="com.google.android.gms.maps.SupportMapFragment"            android:layout_width="match_parent"            android:layout_height="match_parent"            tools:context="mypackage.MapsActivity"/>        <View            android:layout_width="match_parent"            android:layout_height="match_parent"            android:background="#2200ff00"/>        <View            android:layout_width="match_parent"            android:layout_height="20dp"            android:background="@drawable/gradientmap"/>    </RelativeLayout></LinearLayout>

gradientsky.xml

<?xml version="1.0" encoding="utf-8"?><shape xmlns:android="http://schemas.android.com/apk/res/android"       android:shape="rectangle">    <gradient        android:angle="90"        android:startColor="#ffffff"        android:endColor="#00ffffff" /></shape>

gradientmap.xml

<?xml version="1.0" encoding="utf-8"?><shape xmlns:android="http://schemas.android.com/apk/res/android"       android:shape="rectangle">    <gradient        android:angle="270"        android:startColor="#ffffff"        android:endColor="#00ffffff" /></shape>

MapsActivity.java (trivial, but I'm adding it so the example is complete)

public class MapsActivity extends FragmentActivity implements OnMapReadyCallback {    @Override    protected void onCreate(Bundle savedInstanceState) {        super.onCreate(savedInstanceState);        setContentView(R.layout.activity_maps);        SupportMapFragment mapFragment = (SupportMapFragment) getSupportFragmentManager()                .findFragmentById(R.id.map);        mapFragment.getMapAsync(this);    }    @Override    public void onMapReady(GoogleMap map) {        map.getUiSettings().setCompassEnabled(false);        map.setMapType(GoogleMap.MAP_TYPE_NORMAL);        CameraPosition cameraPosition = new CameraPosition.Builder()                .target(new LatLng(39.87266,-4.028275))                .zoom(18)                .tilt(67.5f)                .bearing(314)                .build();        map.moveCamera(CameraUpdateFactory.newCameraPosition(cameraPosition));    }}

The result looks like this:

Another approach could be using a TileOverlay to show a custom map. You can find more info about Tile Overlays here.

To improve the example, you may want to add some parallax effect to the sky so it moves when the map rotates.

Wikipedia arcticle to the creator of Pokèmon Go:

Niantic, Inc. is a software development company best known for developing the augmented reality mobile games Ingress and Pokémon Go. It was formed by Keyhole, Inc. founder John Hanke in 2010 as Niantic Labs, an internal startup at Google.

As you can see, Niantic was an internal startup at Google. Although both companies went their separate way, Google still invests in Niantic. So I think that Niantic has some possibilities, that aren't available for us. As you said, I neither found a way in the Google Maps docs for Android.

I'm not exactly sure if this is true, it's just a guess.