I think of all the alien races I have read about Robert Heinlein’s Martians were truly off the edge as far as being totally alien. This is actually harder to do than most people think. Any author who is using aliens will tell you that writing about something truly alien is hard to do. In order to connect with human readers, there very often has to be something human about the aliens. There has to be some connective point or the reader loses the story. The way Heinlein accomplished this is so subtle with just shades of connection. The main thing is that the humans in the books speak of his Martians as totally alien and requiring people to think differently. It is this learning to think differently that makes the connection.
Most alien races are actually forms of extremism in human behavior or culture. Klingons are representative of human aggression, Vulcans of human reason, Babylon 5’s Narn of deep faith in the face of oppression, etc. In all cases, there is a human elements to alien races. You may make the race totally alien in appearance, but there is usually some element of culture or some behavior that is human. In many ways, the alien in science fiction is a fun house mirror of human behavior.
It calls into question whether or not one can truly do something completely alien that can be understood in literature by human beings. Does the mere fact that a human being came up with an alien race make it in some way human and thus not truly alien in concept? Is it possible to escape this circle and be truly imaginative? Probably not but that does not make it any less fun to try.
At the same time, I also have to ask do we really want to make aliens so outlandish that we can’t learn something from them? It seems that in creating aliens we are trying to say something about ourselves. It in many ways becomes the process of taking out an aspect of our society, culture and/or nature and analyzing it by taking it to extremes. In so doing, it is the author’s hope to pass on some understanding about humans that they can apply to themselves. There is definitely a tension between making an alien different and making it understandable.
Ultimately, we are dealing aliens as a literary tool. How that tool is used depends on the author, but I can’t help wondering how alien are aliens anyway? They are perhaps more of a mirror into the human psyche than anything else I know. That; however, is what makes them interesting.