An (obsolete) meaning of "Bastard" is "Of an unusual or irregular make or proportion";
usually a grade or size in between the (2) most common grades or sizes of an item.
So, A "Bastard file" is a file whose tooth coarseness is of an irregular or unusual proportion,
in between "Coarse" and "2nd. cut" coarseness grades.