It’s the fastest growing genre in the British book world, the one most borrowed from libraries and well-represented in the best-seller lists. Where has the British (and American) love affair with crime fiction come from, and where is it going?

Crime, violence and death have always been major influences in European literature. From the regicide and patricide of Shakespearian tragedy to the breakdown of western society explored in Michel Houellebeqc’s novels; crime, or acts hostile to the social order have been a major source of inspiration for writers. Our fascination with the dark side seems to be inexhaustible.

The roots of crime fiction lie with writers such as Edgar Allen Poe, William Hope Hodgson, possibly even H. P. Lovecraft, all of whom explored a world that lay on the borderline of fantasy and horror. Modern crime fiction is seen as more firmly rooted in the real – but is it?

What is a crime novel? What places one novel on the genre shelves and another on the shelves with the literary novels? Traditional definitions see a crime novel as a book which includes four essential elements: firstly a crime (usually but not always a murder), secondly detectives, professional, private or amateur, thirdly an investigative process and finally the identification of the culprit. But many writers of modern crime fiction play with these expectations, using them, ignoring them and subverting them. A better definition of modern crime fiction might be that a crime novel is a novel with a crime in it, a novel that tells a story and has a narrative thrust.

Modern crime fiction is moving in several directions. One form is less a puzzle and more a sociological drama, focussing on why rather than who, exploring the darker side of the human psyche and the forces that have shaped our society. The focus of the book is on the victim, the bystanders, the investigators, the ways in which the aftermath of a crime spreads its ripples out and changes the life of everyone it touches, rather than the investigation itself. If the heart of crime fiction is the breaking of the social order and its ultimate restoration, these novels address this in less than comfortable ways.

These issues are woven into the more easily recognisable sub-genres: Dreda Say Mitchell and Simon Kernick use the structures of hard-boiled noir; Laura Wilson and Sue Walker write psychological thrillers set both in the present day and the recent past; Ann Cleeves, Francesca Weisman and Simon Beckett play with the conventions of the whodunit, Lesley Horton uses the police procedural to look at the problems of a multi-racial, multi-cultural society.

Contemporary issues are also a feature of this kind of modern crime fiction. Writers are throwing a wider net, exploring the recent past and its reverberations in the current day, moving from traditional settings to international locations. My own most recent book, The Forest of Souls, set partly in Belarus, Stav Sherez in The Devil’s Playground, set in Amsterdam, both explore the way the atrocities of the 1939-45 war echo down the years into the present day; Nick Stone’s Mr Clarinet, set in Haiti, explores this very troubled and unequal society.

These writers can lay a claim to social realism. Their books address current social and cultural issues within the context of the crime fiction narrative, often producing a bleak and chilling commentary on the modern world.

But there is another strand in modern crime fiction that moves away from the real and looks back towards the gothic darkness of Edgar Allan Poe. This sub-genre of the modern crime novel has become obsessed with the psychopath in the form of the serial killer who has replaced the ‘super villain’ of earlier crime fiction. Instead of Moriarty in the Sherlock Holmes stories, Sax Rohmer’s Fu Manchu, Carl Peterson in Sapper’s Bulldog Drummond, we have the Straw Men, Hannibal Lecter, and the Travelling Man.

In his modern incarnation, the super villain has metamorphosed into the serial killer of the modern crime novel. He has become an almost superhuman force for evil, punishment and death and has more in common with the trickster of myth and folklore than the criminal of the traditional whodunit or police procedural. Books in which these things appear lie on the cusp between crime fiction and horror and are frequently graphic and disturbing in their depictions of violence and death.

In the dark and compelling narratives of Thomas Harris, Hannibal Lecter develops, through the three novels in which he appears: Red Dragon, The Silence of the Lambs and Hannibal, to take on godlike qualities and become a Nemesis for the evil and corrupt who operate beyond the power of the law. Mo Hayder, Michael Marshall and John Connolly are all writers who are exploring this field.

In the earliest work that is recognisably crime fiction, the detective as superhero appears. Auguste Dupin is possibly the earliest, and probably inspired Conan Doyle to create Sherlock Holmes. But the detectives of the so-called Golden Age were equally incredible: Hercule Poirot, Miss Marple, Lord Peter Wimsey all solved crimes and caught the villain by methods that were little short of miraculous. Modern crime fiction appears to have moved away from this. The current hero is often deeply flawed – Rebus, Diane Fry, Tony Hill – but despite this the superhero still exists. The powers of detection attributed to forensic scientists in the popular CSI dramas – a TV series that presents itself as gritty and realistic – are firmly rooted in this tradition.

An interesting dilemma for a modern writer would be to pit such a detective against the superhuman serial killer: Sherlock Holmes in pursuit of Hannibal Lecter, for example. This would create a new Laelaps, the inescapable dog, forever chasing the uncatchable Teumessian fox. The gods would have to intervene to solve the dilemma.

But genre fiction does not tend towards innovation. It is popular fiction targeted towards readers who know what they want. The structures remain the same, but the style of writing and the ways in which these structures are represented is what changes. For a mirror of what a society believes, what it hopes for and fears, modern crime fiction may be the best place of all to look.

Carla Banks' new novel The Forest of Souls is published by Harper Collins.