Don't include the leading and trailing / characters when using new RegExp. Those are only used with the literal notation in JavaScript. You also need to escape your backslashes in the string.
db.test.find({ name: { $regex: new RegExp("^((?!nn)[\\s\\S])*$")}});