What is the use of a ...-v21.xml?

I had this themes.xml in the values-v21 folder

A res/values-v21/ directory contains resources that will be used when the device that is running your app is on API Level 21 or higher. If the device is running on an older version of Android, the res/values-v21/ directory will be ignored.

What are they used for? Do they make a difference?

They are used to provide different versions of resources for different versions of Android.

In the case of a themes.xml file, an API Level 21+ device could have a theme that inherits from Theme.Material. However, that theme does not exist on older devices. If you have a theme in res/values/ that tries to refer to Theme.Material, your app will crash on those older devices. So, instead, you put a theme in res/values/ that will work on all devices that you are supporting (e.g., Theme.Holo for a minSdkVersion of 11 or higher), and override that theme in res/values-v21/ to instead use Theme.Material.

You can see that in this sample app, where Theme.Apptheme (my app's theme) inherits from Theme.Holo in res/values/ and inherits from Theme.Material in res/values-v21/. Which version of Theme.Apptheme is used at runtime depends on what version of Android the device has.