Some grammatical properties are extremely stable across sign languages, and arise spontaneously in homesign systems. In particular, signers refer to objects and individuals by creating locations for them in signing space, and sometimes by turning or rotating to these locations to adopt their perspectives. Both the stability and spontaneous emergence of this location-based system suggest that it might be rooted in knowledge that is unlearned, and core to human cognition. We show that non-signers are sensitive to non-trivial constraints on these referential locations when they are realized in gestures (Experiments 1A and 1B), but also when they are realized using non-gestural means, including color, shape, and even sound (Experiments 2A-2D). This evidence suggests that the sophisticated location-based referential system of sign languages might have a cognitive source that is not specific to the visual modality. This raises the question of why spoken languages do not use equivalent systems, which are cognitively available in the auditory modality.