Measuring the Unobserved

Cracking the challenge of measuring unobserved behaviors – drug smuggling, tax fraud, counterfeiting, and illegal immigration – is key to developing better strategies and targeting resources in the right places to catch and deter illegal actions.

Interestingly, observes Dr. John Whitley in a new IBM Center report:  “the local law enforcement community has been a pioneer in measuring and reporting performance and in using these data to drive strategy development and manage execution.”  He points to successes in places like New York City.

But he also observes that” many areas of federal law enforcement do not systematically collect, use, or report basic data on crime rates within their jurisdictions.”  This is in part because federal crimes – like tax evasion or drug smuggling – are less likely to be reported than local crimes – like muggings or break-ins.

Dr. Whitley offers a useful tutorial about measurement and statistics.  He says that “When outcomes are hard to measure, proxy variables are often used,” such as the number of arrests for various crimes.  He notes that “Arrests are an important element in the projection of law enforcement that should be measured, but they are not the outcome.”  In support, he quotes Washington, DC police chief Cathy Lanier: “Arresting people is not a measure of success.  Less crime is a measure of success.”

Five Methods for Estimating Unobserved Events

Dr. Whitley says that, in order to “effectively manage federal law enforcement activities, officials and policy-makers in charge must have an idea of what is happening.”  He describes five statistical methods for estimating data on unobserved events:

Method 1:  Use Administrative Records.  Agencies collect reams of administrative data and this should be the first place for analysts to look.  For example, the US Border Patrol collects information of all apprehensions of illegal border-crossers. Analyzing how many of these people try to cross again can provide insight into which deterrence strategies work better than others.

Method 2:  Use Surveys.  Surveys can be costly, especially if they have to be statistically valid for small geographic areas.  And surveys can be suspect, especially if there are concerns about the truthfulness of respondents.  But using existing surveys, such as the National Crime Victimization Survey, can be a cost-effective way to understand how to understand and measure potential undercounting.

Method 3:  Inspections, Investigations, and Audits.  Typically, law enforcement prioritizes its investigations.  But if this approach is to be used to estimate the impact of law enforcement, it would need to be done randomly in order to be statistically valid.  This would not make sense for the most part, but if employed in a limited way, it could provide valuable data.  A good example is the IRS’s National Research Program which does annual samples of 13,000 taxpayers and is used to estimate the size of the tax gap – the amount of taxes not being paid.

Method 4: Experimental Methods.  Adding or modifying routine law enforcement activities can help estimate crime rates, as well.  For example, Customs and Border Protection conducts random secondary screening of passengers at ports of entry (e.g., the border or airport) to evaluate the effectiveness of primary screening processes.

Method 5:  Technical Measurement. Well-known methods of technical measurement include red-light and speeding cameras.  The Secret Service also uses technical measures to identify counterfeit money

Conclusions

Dr. Whitley cautions that getting the measures right is critical.  If the wrong measures are selected, then it could result in decision-makers making bad decisions or wasting money.  But he says the law enforcement community has some inherent advantages because their performance management systems are “relatively straightforward,” even though technical challenges exist because “outcomes are often unobserved and cannot be directly measured.”  But if approached thoughtfully, “it is possible to bring the radical reforms that have been seen in state and local law enforcement to the federal level.”