The approximate hand-waving description:
The AFCI circuit has a sensor which measures the actual pattern of current flow. Think about what an oscilloscope does, but rather than creating an image for a human to look at, the measurement is loaded directly into a small computer.
This computer is supposed to look for specific patterns of current flow associated with arcs. The better the job of pattern recognition, the better the AFCI will differentiate between arcing faults and normal operation.
Different loads have different operating characteristics and different current flow patterns. An arcing fault will look quite different than a resistive load or an induction motor. Unfortunately, some loads have totally normal current flow patterns that look quite a bit like arcing faults. These loads make the job of differentiating between faults and normal operation quite difficult.
If your computer has too broad a definition of the current flow pattern that looks like an arcing fault, then you will have lots of nuisance tripping. If the definition is too narrow, then your breaker will miss real arcing faults.
Until recently, this has been so much of a problem that only current flow that looks like an arc _and_ is well in excess of the breaker trip rating will be treated as a fault. This has the benefit of speeding up the response to an arcing short circuit, but this approach cannot respond to 'series arcs', meaning faults where the current passes through an arcing connection and then through the balance of the load.
As of January 1, 2008, so called 'combination' AFCIs will be required. These have to properly detect arcing down to 5A, and should be able to detect arcing faults.
It remains to be seen if manufacturers can actually deliver a product which can both detect series arcs _and_ not have excessive problems with nusiance tripping.
-Jon