The origin is unclear. It can be traced back prior to 1583 where its first recorded use was attributed to a sermon by John Calvin. It is probably older.
The phrase "neither here nor there" is from Arthur Golding's translation of "The sermons of J. Calvin upon Deuteronomie" in 1583. It's thought to originate in the 16th century.
