Financial audits, including healthcare, tax, and due diligence audits

- Is a set of of asset accounts (e.g., accounts receivable) materially
*overstated*? By how much? - Is a set of of liability accounts (e.g., accounts payable) materially
*understated*? By how much? - Of a collection of bills that were written off, what fraction of the value was written off in compliance with applicable policy?
Is a set of bills for healthcare

*overstated*? By how much?

Health Care Fraud Prevention and Enforcement Action Team (HEAT)

Since 2007, >1,400 defendants collectively falsely billed Medicare >\$4.8 billion. In 2011, HEAT federal health care fraud takedown involving $530 million.http://www.hhs.gov/news/press/2014pres/05/20140513b.html http://www.hhs.gov/news/press/2013pres/05/20130514a.html

- Is a set of of asset accounts (e.g., accounts receivable) materially
Election auditing

- Was the margin for the reported winner
*overstated*? By how much? - Did the reported winners really win?
- Were the correct numbers of seats allocated to each party?

- Was the margin for the reported winner
Damages in litigation

- Were employees paid all the overtime they were due? If not, how much is owed?
- Were employees given all the meal and rest breaks they were due? If not, what is owed?
- Were mortgage origination fees "padded"? By how much?
- Was a patent infringed? If so, how much is due in royalties or license fees?
- In a collection of mortgages, what fraction of the original balances were in loans that did not comply with the applicable underwriting guidelines?

Causal inference in experiments, marketing, etc.

- Does a tool for targeted advertising work? How much does it increase sales?
- Does a purported remedy work? How large is its effect?

All these can be reduced to inferences about the mean of a nonnegative or bounded finite population.

That is, there are:

- known numbers $N > 0$, $ \{\ell_j\}_{j=1}^N$, and $\{u_j\}_{j=1}^N$, with $\ell_j \le u_j,\; \forall j$
- an unknown population of numbers $\{x_j\}_{j=1}^N$ that satisfy $\ell_j \le x_j \le u_j,\; \forall j$

We want to draw inferences about $$\mu \equiv \frac{1}{N} \sum_{j=1}^N x_j.$$

We want more than a point estimate:

- rigorous hypothesis tests
- exact or conservative confidence intervals
- generally, 1-sided confidence interval: lower confidence bound or upper confidence bound, depending on context.

Nonparametric confidence bounds for population percentiles require only a random sample. (They can be constructed by invert binomial or hypergeometric tests; see, e.g., Lehmann.)

Nonparametric confidence bounds for the *mean* require prior information, because a single item
can have an arbitrarily large influence on the population mean.
Unless the distribution is symmetric, the mean need not equal the median.
In many applications, the mean (or population total) is relevant but the median is not.

For instance, in both financial and election auditing one might like to determine from a random sample whether a reported total differs materially from the total that a full manual audit would show.

In auditing, the population of errors typically consists mostly of zeros or small values, but occasional large values are possible, which makes the normal approximation inadequate. The mean error in the population determines the total error; the median error is not directly relevant.

Most of the methods we consider will assume that we are sampling *with* replacement. There are three main reasons:

- In the applications contemplated, the sample size is a small fraction of the population size, so there is not much difference between sampling with and without replacment.
- The math is simpler for sampling with replacement.
- In great generality, uncertainties for sampling with replacement are larger than for sampling without replacement (see Hoeffding); hence, these results give a conservative impression of what is possible.

Many of the approaches do have extensions to sampling without replacement, but the bookkeeping is more complicated.

Many of the methods we will consider draw samples with replacement, but not with equal probabilities. In particular, selecting items with probability proportional to a bound on "size" turns out to be very helpful.