Tastes kind of like hay smells, if that makes any sense. It's expensive because it takes close to ten thousand plants to get even a little of it. It's the dried stigma of the saffron crocus plant.
I personally find saffron to have a very mild sweet taste, although many others refer to it as being "earthy". It is expensive due to the small quantities produced per plant and the high amount of labor required to cultivate it.