Using Cgroups to limit cpu usage
cpu cgroup is work conserving, ie. a task would not be stopped from using cpu if there is no competition. If you want to put a hard limit on amount of cpu a task can use, try setting cpu.cfs_quota_us and cpu.cfs_period_us.
Look at the documentation here.