H.P. Lovecraft is reputed to have disliked movies, but that's not strictly true. He enjoyed adventures and comedies and wrote a poem about Charlie Chaplin, but he cared little for the horror films that he'd seen and hoped that his own stories would be spared the same treatment. It's as though he could see what the future had in store. Maybe he noticed something about the way stories were adapted and realized that the screen would not be able to capture the spirit of his writing. Decades later, Lovecraft's body of work is notorious for its lack of success in being adapted for movies. There are many reasons for this.
Lovecraft's audience represents a cult following, and Hollywood economics require that a movie be marketable to a mainstream audience. The better adaptations tend to come from independent studios that don't have to recover large budgets and are free to target smaller, devoted audiences.
Obscure, intellectual stories are not viable for mass distribution. The old saying was, "will it play in Peoria?" These days, one might imagine it sitting on a Wal-Mart shelf between a teen slasher and a creepy ghost story. Will the average shopper even pick up the box to look at it? These are the questions asked by studio executives when they decide which movies are going to be produced. If they don't buy into the concept, nobody will get the funding. Dagon was unmade for fifteen years because executives felt that "fish are not scary." This issue, combined with a high demand for his services, has kept Guillermo del Toro's long-rumored At the Mountains of Madness from getting made.
Audiences need familiar plot devices to help guide their understanding. The Cthulhu Mythos has its own set of familiar elements, but they're only understandable to audiences who are already knowledgeable about the Cthulhu Mythos. Jason with a machete: that's easy to understand. The Cthulhu Mythos needs to be explained, and this requires a lot of potentially wordy expository dialogue. You can name-drop the Necronomicon, the city of Arkham and Miskatonic University and Lovecraft fans will understand and appreciate it, but neophyte viewers need some kind of introduction. This often leads filmmakers to incorporate standard horror themes that dilute Lovecraft's ideas.
Dagon solves this problem by focusing on the present situation instead of its relation to the larger Mythos universe. Its source story depends on historical descriptions, but Dagon makes these largely irrelevant by casting its protagonist as a foreigner who cannot speak the language and has no knowledge of the location. The Dunwich Horror (1970) is less successful in its attempt to incorporate elements of Satanism that would be familiar to a post-Rosemary's Baby audience.
Lovecraft was a lover of antiquity. Never mind that his stories were written nearly a century in the past, he favored language that was even then archaic. This creates a perceived need to modernize his stories for the screen. That can be effective if done well, but the results can be embarrassing.
The stories that are most frequently and successfully brought to screen are those from his middle period. This is where he moved away from the gentle fantasy of his earlier stories and toward a more standard, yet original, type of horror. These stories feature a Poe-like sense of the macabre mixed with a darker fantasy element. This new form of fantasy is what would mature into what we now call the Cthulhu Mythos. The deep, cosmic terror that is so valued by Lovecraft's fans had not yet developed, and the stories from this era take place on a more personal and visual level. They are good candidates for filming but tend to be very short. These often wind up as episodes of television series like Night Gallery or as segments in anthologies.
"Herbert West: Reanimator" is from this period. Re-Animator is frequently mentioned as among the best Lovecraft film adaptations, but it's not very representative of his work and is not even a story of which Lovecraft was particularly proud. "The Lurking Fear" has been attempted several times. The Descent is not explicitly stated as a Lovecraft adaptation, but it is essentially "The Lurking Fear." "Cool Air" and "Pickman's Model" are other stories from this period that commonly appear in television shows and anthology films.
Lovecraft's cosmic horrors don't translate well to a visual medium. Film makes things explicit, and this is rather opposite to how Lovecraft's stories work and what makes them effective. The terrors described by Lovecraft are often invisible or unimaginable. If you depict such a thing, you limit it to what your own imagination is capable of, not to mention your skill at making this image real. What if your special effects department is not up to the task? Even a filmmaker who can successfully bring their vision to life will only have appeal for the audiences who share that vision. To truly channel Lovecraft, you have to engage the audience's imagination.
A common element in Lovecraft stories is the appearance of things from dimensions that obey different sets of physical laws. How does one depict a building whose architecture violates the laws of geometry? Escher stairs might be a good start, but filmmakers who even bother to tackle that concept typically settle for putting crazy angles and curves in the buildings. It looks weird but is not even close to the intended concept. At one point in The Call of Cthulhu, a character simply disappears. A viewer unfamiliar with the story would have no idea that the character had fallen into an angle that violated the laws of mathematics.
A similar example would be "The Colour Out of Space," which features an object of a color never before seen by mankind. How do you depict a color that doesn't even exist? Die, Monster, Die used a misty green, radioactive glow. That's a way to show that something is there, especially given a small budget in a pre-computer era, but it doesn't produce the original effect.
The Dunwich Horror (1970) uses then-fashionable psychedelic effects to express the idea that something mind-blowing is happening. That now looks gimmicky and dated but is very much in line with Lovecraft's ideas. It may, however, only make sense to those whose minds have previously been blown. The Unnamable takes a particularly lazy approach: it depicts its central terror as a kind of composite monster. It's weird and is something for which we don't yet have a name, but there's nothing inherently unnamable about it. The Thing (1982) was inspired by although not based on Lovecraft, and it is completely successful at depicting an unnamable terror. Its visual appearance is shocking and otherworldly, but its true horror is based on its nature rather than its appearance.
Many of Lovecraft's stories involve gigantic ruins of enormous forgotten cities. Budget becomes an issue in representing these. When a movie is already targeted at a smaller, niche audience, there's no way that it can sell enough tickets to finance such expensive set designs. Even CGI is not cheap, and it's not the solution to everything.
Lovecraft's stories are very "talky," which is to say that they are mostly description with little dialogue or action. Their narratives take place very much in their protagonists' minds. It is very challenging to find a way to draw this inner world of thought out to a visual, external world. Because of this solitary nature, there's little character interaction. We can remedy this by building new characters into the story, but we've now taken many liberties and may already have strayed too far from the original. It nevertheless has to be done, as few of Lovecraft's stories have female characters. That gives half of your potential audience nobody onscreen with whom to relate. The other half of the audience is unenthusiastic about watching a movie that has no women.
Some stories that might film well have never been attempted. "The Rats in the Walls" has an uncinematic storyline and set requirements that would once have been prohibitive. It might be achievable in this age of CGI and could be an awesome picture if done right. "The Shunned House" is the chilling tale of a kind of psychic vampire and is something to which audiences could relate. Many of his earlier fantasy works, such as The Dream Quest of Unknown Kadath, would work well as animated features.
John Carpenter mentioned several of the above factors as reasons why adapting Lovecraft is so difficult. The result of all this is that bringing H.P. Lovecraft's stories to the screen is a difficult task that is unlikely to please moviegoers beyond his relatively small cult of fans. An unsuccessful attempt is likely to please nobody and will lose boatloads of money. Maybe it is best that we avoid raising our hopes too high and remain grateful for the few instances that work.