In the census in 1921 there were over 150000 British Subjects in India. Over 40000 of them were women, so we are not talking just army.

In 1891 there were 238,409 with english as their mother tongue (before the census broke this out).