I remember a classmate once arguing that "English" was a misnomer for what we do. She suggested that "Literature" would be a much better term, since literary students will find themselves faced with other languages and translations all the time. Heck, the most epic literary works we know were written in Greek and we deal with translations (except the semester I spent translating Book 6 of the Aeneid for advanced Latin, and also all my coursework in Anglo Saxon). But if "English" is a term fraught with peril, I'd say "Literature" is equally bad if not worse. What about the linguistic side of our studies? Surely that deserves some recognition.
Granted, many people I know assume that English is all about reading books. And honestly, you really don't always need a primary source (in the traditional sense of the word) (also, I love ITED). My MA research has no primary literary text, strictly speaking. But that's where "English" becomes a hazy label that I can't easily explain to civilians. The best I can do is to say that it's one of many angles that Humanities scholars can take, though I think I'll always have trouble explaining what literary theory is, and why. Perhaps the boundaries between the different departments are dissipating as time goes on, particularly with shifts in media.
On the other hand, for many years I've just taken it for granted that Academia accepted that we all "read" our environments, media, etc. every day of our lives. Trying to explain what my master's essay was about to anyone is truly painful, yet people unfortunately find it necessary to keep probing after I say "well it would be totally meaningless/uninteresting to you."