Ask those people why they say this.  Have they tried to learn to speak isiXhosa,  and if so, for how long?  How motivated were they?  Which language-learning program did they follow?  With the right mental approach and the right learning package, nothing is impossible.