Subtle vibrations take after vibrations of sound and wavelengths of light. The stone isn't literally, physically vibrating, it just means that the stone has a naturally high spiritual vibration (or frequency, if you want to term it that way). Think of the physical plane as the densest, the etheric plane as middle ground, and the astral plane as the least dense (if you have a belief system that reaches higher/less density planes, then those would be the least dense of vibrations.
Generally stones associated with psychic awareness (such as amethyst), "booster crystals" (such as quartz), or stones associated with higher beings (selenite for angels, for example; stones associated with deities will be varied and often personal to the devotee/practitioner) are considered "high vibrating stones" as the spell calls for.
Crystals such as smokey quartz, tiger's eye, and hematite tend to be associated with the physical body, therefor would be considered by this paradigm as a lower vibration. However, that does not make them lesser in anyway.
Keep in mind that these associations tend to be more New Age and personalized to some degree. If you find differently, then great. It won't mean you're wrong. If you find this clicks, also great; the goal is to make things work for you.