One school has unlocked the code to getting more students to take on harder maths subjects.