Anti-virus software typically uses two different techniques to accomplish this:
- Examining files to look for known viruses by means of a virus dictionary
- Identifying suspicious behavior from any computer program which might indicate infection
Most commercial anti-virus software uses both of these approaches, with an emphasis on the virus dictionary approach.
Virus dictionary approach
In the virus dictionary approach, when the anti-virus software examines a file, it refers to a dictionary of known viruses that have been identified by the author of the anti-virus software. If a piece of code in the file matches any virus identified in the dictionary, then the anti-virus software can then either delete the file, quarantine it so that the file is inaccessible to other programs and its virus is unable to spread, or attempt to repair the file by removing the virus itself from the file.
To be successful in the medium and long term, the virus dictionary approach requires periodic online downloads of updated virus dictionary entries. As new viruses are identified "in the wild", civically minded and technically inclined users can send their infected files to the authors of anti-virus software, who then include information about the new viruses in their dictionaries.
Dictionary-based anti-virus software typically examines files when the computer's operating system creates, opens, and closes them; and when the files are e-mailed. In this way, a known virus can be detected immediately upon receipt. The software can also typically be scheduled to examine all files on the user's hard disk on a regular basis.
Although the dictionary approach is considered effective, virus authors have tried to stay a step ahead of such software by writing "polymorphic viruses", which encrypt parts of themselves or otherwise modify themselves as a method of disguise, so as to not match the virus's signature in the dictionary.
Suspicious behavior approach
The suspicious behavior approach, by contrast, doesn't attempt to identify known viruses, but instead monitors the behavior of all programs. If one program tries to write data to an executable program, for example, this is flagged as suspicious behavior and the user is alerted to this, and asked what to do.
Unlike the dictionary approach, the suspicious behavior approach therefore provides protection against brand-new viruses that do not yet exist in any virus dictionaries. However, it also sounds a large number of false positives, and users probably become desensitized to all the warnings. If the user clicks "Accept" on every such warning, then the anti-virus software is obviously useless to that user. This problem has especially been made worse over the past 7 years, since many more nonmalicious program designs chose to modify other .exes without regards to this false positive issue. Thus, most modern anti virus software uses this technique less and less.
Other ways to detect viruses
Some antivirus-software will try to emulate the beginning of the code of each new executable that is being executed before transferring control to the executable. If the program seems to be using self-modifying code or otherwise appears as a virus (it immeadeatly tries to find other executables), one could assume that the executable has been infected with a virus. However, this method results in a lot of false positives.
Read full here: http://www.antivirusworld.com/articles/antivirus.php