A:

Would you stop cracking your knuckles?!

B:

Oh, sorry. Why, does it annoy you?

A:

I'm not sure, but the sound just gets on my nerves!

get on someone's nerves ：　誰々をイライラさせる