I have a similar problem with my CSR. The starter seems to really struggle turning the engine over. After saying that she always starts on the button.
I do agree with Michaels comments. One other possibility to me is a tight engine. When my engine gets very hot it seems tighten up and can stall. I have to kick her then as the button has no effect.
I intend to strip her down this winter and investigate the issue.
I will also be overhauling my starter motor and giving her new bushes and brushes etc