Grab it here.

Related posts:

1. Update to Functional Equations Reference
2. Update: Reference for Functional Equations
3. Difference and Functional Equations Reference

Related posts:

1. Update to Functional Equations Reference
2. Difference and Functional Equations Reference
3. Update to Functional Equations Reference (version 1.3)

I finally laid my hands on Donald Knuth’s The Art of Computer Programming (what a wonderful set of books!), and found a neat algorithm for generating random integers 0, 1, 2, … , n – 1, with probabilities p_0, p_1, … , p_(n-1).

I have written about generating random numbers (floats) with arbitrary distributions for one dimension and higher dimensions, and indeed that method can be adapted for generating integers with specific probabilities. However, the method described below is much more concise, and efficient (I would guess) for this special case. Moreover, it is also easy to adapt it to generate floats for continuous distributions.

Description of the Algorithm

The basic idea of the algorithm is simple. We have two tables of length n that contains integers (K, L), and a third table that contains probabilities (P). The first table merely contains the integers 0 to n-1 (thus, we need not actually store it explicitly). The other two tables are computed before generation (more about that below).

To generate a random number, we generate a random integer (uniformly distributed between 0 and n-1 inclusive), and a random float (uniformly distributed between 0 and 1). The integer tells us which cells in the table to use. First we lookup the probability in P at that index. If the float is smaller than that value, we return the integer in K, otherwise we return the integer in L. (Note, the integer in K is exactly the random integer itself. Therefore, we do not actually have a table K – we simply use the random integer.)

Now this will only give the desired result if the tables have been constructed for this to work. Before looking at how these tables are generated, let us look at a very simple example. Suppose we want to generate 0, 1, 2, with probabilities 3/18, 7/18, 8/18. Can you see why the following tables will work?

There is only one way to generate 0: if the random integer is 0, and the random float is below 1/2. This will happen with probability 1/3 x 1/2 = 1/6 = 3/18.

There are two ways of generating 1:

• if the random integer is 1 (probability 1/3 = 6/18); or
• if the random integer is 2, and the float is above 5/6 (probability 1/3 x 1/6 = 1/18).

Adding these probabilities, we get 6/18 + 1/18 = 7/18.

There are also two ways to generate 2:

• if the random integer is 0 and the random float is above 1/2 (1/3 x 1/2 = 1/6 = 3/18); or
• if the random integer is 2 and the random float is below 5/6 (1/3 x 5/6 = 5/18).

Adding these probabilities, we get 3/18 + 5/18 = 8/18.

Generating the tables

Consider this problem:

• We have n squares that we want to paint.
• We have five colours of paint, possibly different amounts of paint for each colour.
• Each square has a border in one of the colours; no two borders are the same colour.
• In total, there is just enough paint to cover the n squares exactly.
• We want to paint each square with at most two colours, with one colour matching the border.

To do this, we sort the paint buckets in ascending order. We paint the square that matches the first bucket with the first bucket, and whatever remains with the last bucket. The first bucket is empty (why?), and there might be some paint remaining in the last bucket. We now put this bucket back so that the buckets are sorted according to the new quantities of paint. The first square is completely covered (why?). Note that the painted square’s colour corresponds with the depleted colour.

The situation is now: we have n-1 unpainted squares, and n-1 colours of paint. This is the same problem as the initial problem, with one less square and one less colour. Therefore, we proceed as before, and repeat this process until all the squares have been painted and all the paint has been used.

To answer the two why’s above:

The first bucket is always empty, because the smallest bucket cannot cover more than one square. Since it is the smallest bucket, all other buckets must contain at least as much paint. Thus, we have n colours, and enough paint of each colour to paint more than one square. Thus, in total, we must have more paint than is required to paint n squares. But we said that we have exactly the right amount of paint needed, not more. Therefore, the smallest amount of paint cannot cover more than one square.

The first square is always covered completely, since the last bucket always contains enough paint for at least one square. If it did not, since it is the largest bucket, all other buckets will paint less than one square. In total, we would have n colours, each that can cover less than one square. Thus, all our paint will cover less than n squares: we do not have enough paint. But we said that we do, so this cannot be. Therefore, we must have enough paint in the last bucket to cover at least one square.

Below is an illustration of this process for three colours. Here we assume that 1 litre of paint covers 1 square. We have 1/3 = 3/6 litre of red, 7/6 litre of green, and 8/6 litre of cyan.

Sort Paint

Paint from first bucket

Paint from last bucket

Sort Paint

Paint from first bucket

Paint from last bucket

Sort paint

Paint from first bucket

As you can see, the solution above corresponds with the tables given in the example above. Indeed, the algorithm for calculating the tables is exactly the same as the paint algorithm:

• The n colours correspond to the n integers (from 0 to n-1) that we want to generate.
• The initial amounts of paints corresponds with the (relative) probabilities that we want to generate each integer.
• The amount of paint used to paint a square of the same border is the entry in table P – the probability of using the number associated with that cell (i.e., “the border”).
• The other colour used to paint a square (if any) corresponds to the entry in table K.

A Small Optimisation in the Implementation

Generating two uniform random numbers for the price of one

There is a trick to generate a random integer (0 <= n < k) and a random float (0 <= x < 1) from a single random float (0 <= u < 1) that is used in the implementation of this algorithm (see download below).

The trick is:

• n = floor (uk)
• x = uk – n

This assumes things about the random generator and the accuracy required. (I do not want to get into the details here).

Download

A python implementation of the above algorithm.

non_uniform_random_int.py

Related posts:

1. Generating Random Points from Arbitrary Distributions for 2D and Up
2. Generating Random Numbers with Arbitrary Distributions
3. Estimating a Continuous Distribution from a Sample Set

I will cover the case for 2D sets here, but the ideas are easily extended to any dimension.

Visual Inspection From a Scatter Plot

The easy way to estimate the distribution is to simply look at a scatter plot of the samples.

	A scatter plot of a 2D sample set. (A pixel was simply drawn at each point of the sample. The images was slightly blurred to give the pixels more substance).Here we can already estimate the distribution, as we can intuitively “see” the shape.

If the distribution is simple, we can often “guess” suitable parameters for a mathematical function. But this method is not suitable when more accuracy is required. And obviously this method does not work for higher dimensions.

I always use a visual inspection as a first step when using the other methods described here. First, it allows you to decide which approach to use. Second, it serves as a rough benchmark to test the results of a better method against.

Averaging Over a Grid

For this method, we divide the domain in cells, and count the number of points in each cell. To normalise, we divide the counts by the total number of points to get a distribution.

Typically, we choose a cell size large enough so that all cells contain a few points.

It is possible to use different cell sizes, so that regions of higher density has more cells, for instance, by using a quad tree. However, this approach makes it difficult to interpret the values (they should be scaled down by a factor of the area they represent), and to generate random numbers from this distribution. I can’t really imagine a situation where this approach will be preferred above using a regular grid or convolution.

Convolution

There are two ways to implement convolution. The first should be used when the sample set is small relative to the size of the domain, otherwise the second method will be more efficient.

Sparse Point Convolution

Generate a very granular empty grid over the domain.

For each point p in the sample set,

• map p to the grid (calculate the coordinates x,y of p in the grid),
• add a number to all the cells in the neighbourhood of p in the grid.

Now normalise the grid.

The neighbourhood is typically a circle or a square. The number you add can be the same for all neighbours (in which case any positive number will do), or it can be scaled depending on the distance from p.

The size of the neighbourhood depends on the density of your sample. In general, the denser it is, the smaller the neighbourhood can be. Smaller neighbourhoods lead to faster execution. To prevent “holes” in the approximation, use a radius that corresponds to the maximum of the distances between a point and its closest neighbour.

	Estimation with a constant square neighbourhood.
	Estimation with a constant circular neighbourhood.
	Estimation with a circular neighbourhood with a falloff.

Traditional Convolution

Generate a very granular empty grid over the domain.

For each point p in the sample set, map p to the grid (calculate the coordinates x,y of p in the grid), and add one to that location in the grid.

Now choose a square, symmetrical convolution matrix, and perform a discrete convolution on the grid:

new_grid[i, j] = sum_{i,j} grid[i][j] * c[i][j]

Here the i, j go over the indices of the convolution matrix. (Normally, the convolution is defined as new_grid[i, j] = sum_{i,j} grid[n - i][n - j] * c[i][j]. However, since we are suing a symmetrical matrix, these definitions are equivalent, and we need not perform the extra calculation).

Now normalise the grid.

The convolution can be a square or circle of 1s, or be filled with numbers that grow smaller outwards. These correspond to the three neighbourhoods described for the sparse convolution.

Note that the new_grid is larger than the original by one less than the size of the convolution matrix in each dimension. The centre of the new grid corresponds with the original grid.

For example, if the original grid was 100×100, and the convolution matrix was 5×5, the new grid will be 104×104. The point (2, 2) in the new grid corresponds to point (0, 0) in the original grid.

About Normalisation

It is customary to normalise the distribution so that all the probabilities add to 1. But for many purposes we need only relative probabilities, and this step can be skipped. For example, the method of generating random numbers described in a previous post uses only relative probabilities.

A Few Tips

• Always test your distribution by generating a random set from it, and comparing it with the original sample. They should match qualitatively.
• The smaller your sample set, the cruder the approximation should be. That is, cells, neighbourhoods or convolution matrices should be big. There is a limit to the accuracy you can obtain from any sample – if you try to exceed it, your results will be poor.
• The most common implementation errors are made at the borders (of any grid or matrix) – watch out for them!

Download

There is an example of implementation in 2D in with the Python Image Code. See the file random_distributions_demo.py.

Related posts:

1. Generating Random Points from Arbitrary Distributions for 2D and Up
2. Generating Random Integers With Arbitrary Probabilities
3. Generating Random Numbers with Arbitrary Distributions

I have already covered how to generate random numbers from arbitrary distributions in the one-dimensional case. Here we look at a generalisation of that method that works for higher dimensions.

The basic trick, while easy to understand, is hard to put in words (without reverting to mathematical equations). For two dimensions, we divide the plane into slices. Each slice is a 1D distribution. We also calculate a distribution from summing the frequencies in each slice. The latter distribution gives us one coordinate, and the appropriate slice to use. The distribution of that slice then gives the second coordinate. All distributions are put into inverse accumulative response curves as was done to generate one-dimensional random numbers. (You should review that before implementing the 2D case).

In more dimensions, we also slice the space up into 1D distributions. Sums of these give us more distributions, which we can sum again, and again, until we reach a single distribution. This is used for the first coordinate, and to determine which distribution to use for the next coordinate. This goes on, until a 1D slice gives us the final coordinate. Again, all distributions are converted to inverse accumulative response curves.

If the above is unclear, I hope the detailed description below clears things up.

Two Dimensions

Describe the Distribution

Divide the domain into a regular grid. Let us use the domain 10..40 x 10..40. This means, all the points we will generate will fall in the square between the lines x = 10, x = 40, y = 10 and y = 40.

For later use, we will need the minimum x-coordinate of the domain (x0 = 10), the width of the domain (w = x1 – x0 = 40 – 10 = 30).

Assign to each cell the number of random numbers that should be generated for that cell for any suitable total of points. You need not normalise these values.

For our example we will use the 4×4 grid shown below:

For later use, we need the width of the grid (gw = 4).

The method described here won’t work if any of these values are 0.

Calculate Cumulative Grids

Calculate the cumulative sums for the columns of the array.

Calculate a cumulative sum for the last row

Construct Inverse Response Curves

For each column of cumulative sums, append a zero at the beginning, and create an inverse response curve as described for the one dimensional case. Thus the first columns inverse response curve will be created from 0, 1, 3, 7, 15. Call these curves cy[0] … cy[3].

For the row of cumulative sums, append a zero at the beginning, and create an inverse response curve. Call this curve cx.

Determine Random Point

You are now ready to generate random points from the specified distribution.

First, generate two uniform random numbers, urx and ury.

Use urx to do a lookup into curve c[x]. The result rx gives the x-coordinate of your non-uniform random number. Use this number to decide which column curve to use: subtract the minimum domain x-coordinate, divide it by the domains width, multiply it by the number of columns, and floor it to an integer.

ix = (rx – x0)/w * gw

Now use ury to do a lookup in cy[ix]. The result ry is the y-coordinate of the number.

This image shows a plot of samples generated from the distribution specified above (normalised to fit in the image boundaries). Brighter areas indicate that more points occur in that region.

Visually, it looks like the sample mimics the original distribution well.

N Dimensions

Describe the Distribution

Divide the domain in a regular N-dimensional grid, and assign frequencies to cells in the grid. For future calculations, we will need the lowest coordinate of the domain along each dimension (x0_k), the width of the domain for each dimension (w_k), and the number of cells in the grid for each dimension (gw_k).

Calculate Cumulative Grids

Accumulate sums along one dimension in a N-dimensional grid G_N. The last subgrid contains the totals of all “columns” along that dimension, and is a (N-1) dimensional structure.

Accumulate sums of this subgrid into a (N-1) dimensional grid G_(N-1) along another dimension. Again, the last subgrid contains the totals of the columns of that dimension.

This must be repeated until a single row, G_1 is produced. In general, the last subgrid in G_k contains the totals of columns in G_(k-1) along dimension k. This must be accumulated in a (k-1)-dimensional grid G_(k-1).

Construct Inverse Response Curves

Now for each grid G_k, we need to construct inverse response curves from the columns along dimension k. Remember to append 0 at the beginning of each column. The curves must be put into a k-1-dimensional structure, C_k.

Determine Random Point

Generate N uniformly distributed random numbers ur_1…ur_N.

Use ur_1 to do a lookup in C_1. The result r_1 is the first coordinate of your point. Use it to determine to calculate an appropriate index i_1:

i_1 = (ur_1 – x0_1)/w_1 * gw_1

i_1 is the index of the curve to use from C_2: lookup ur_2 in C_2[i_1] to obtain r_2. This is the second coordinate of your point. Determine i_2:

i_2 = (ur_2 – x0_2)/w_2 * gw_2

i_1 and i_2 determine the curve to use from C3, i.e. the curve C3[i_1, i_2].

Repeat this process until all coordinates are determined. In general, use ur_k to do a lookup in C_k[i_1, i_2, …, i_(k-1)] to obtain r_k. Use this to calculate i_k:

i_k = (ur_k – x0_k)/w_k * gw_k

That’s it!

Download

There is an example of implementation in 2D in with the Python Image Code. See the file random_distributions_demo.py. (I know it is annoying that it is coupled with all the other image code… I am working on a better solution!)

Related posts:

1. Generating Random Numbers with Arbitrary Distributions
2. Generating Random Integers With Arbitrary Probabilities
3. Estimating a Continuous Distribution from a Sample Set

Related posts:

1. Update: Reference for Functional Equations
2. Update to Functional Equations Reference
3. Update to Functional Equations Reference (version 1.3)

For many applications, detailed statistical models are overkill. Instead, we can get away with a rough description of the distribution – not in mathematical formula form, but just as a graph with a few sample points.

For example, when trying to model the traffic around a school, you might know that the graph looks something like this:

The input is the number of minutes before the first bell rings, and the output the number of children dropped off at that time. You know that most kids are brought before the bell rings, and that the closer to the bell, the more kids are being brought every minute. Only a few kids are late.

This tutorial describes how to generate random numbers that can generate a distribution described by an arbitrary (piece-wise linear) curve, as the one above.

Implementation

1. Calculate an accumulative probability

The first step is to run an accumulative sum of our original sample points. The following code snippet shows how it can be done in C++; the array “samples” contains the original samples.

for(int i = 1; i < n; i++)
{
    samples[i] += samples[i - 1];
}

  The snippet above does the calculations in-place, but it is not necessary, as is shown in the snippet below:

newSamples[0] = samples[0];

for(int i = 1; i < n; i++)
{
    newSamples[i] = samples[i] + newSamples[i - 1];
}

  The graph above shows the accumulative probability density.

2. Calculate inverse sample points

This step is easy – we simply swap the input and output samples. It is not necessary to do this explicitly, just swap them in the argument list of the function call in the next step.

3. Interpolate between samples

I use a special data structure that makes it very easy to compute the interpolations from discreet sample points: the response curve. There are two varieties. The ordinary response curve, with uniformly spaced samples, and the xy-response curve, where samples can be arbitrarily spaced. Ordinary Response Curve. XY-Response Curve. The implementation is very simple, so I won’t describe it here. A C++ implementation is available in the Special Numbers Library; a Python implementation is available with the example below. Here I simply explain the usage as it relates to this tutorial. We proceed as follows:

• First, we construct an xy-response curve from our samples of the inverse accumulative probability.
• Then we sample this curve at regular intervals, and load it into a ordinary response curve (we do this, simply because the ordinary response curve is much faster than the xy-version).

//...

int oldSampleCount = 7;

int newSampleCount = 20;

float inputMin = 0.0f;
float inputMax = 3.2f;

//note: input and output is swopped arround, because we want the inverse!

XYResponseCurve<float, oldSampleCount> xyCurve(outputSamples, inputSamples);

for(int i = 0; i < newSampleCount; i++)

{

    input = ((float) i / (newSampleCount - 1)) * (inputMax - inputMin) + inputMin;

    uniformOutput[i] = xyCurve(input);

}

ResponseCurve<float, newSampleCount> curve(inputMin, inputMax, uniformOutput);

4. Map uniform random numbers to the input range of the IAPDF, and calculate the output

Now that we have the response curve, we can map uniform random numbers to the appropriate input range. For the example above, we need to map to the range [0, 3.2]. The snippet below shows how to generate random numbers with the distribution shown above:

//...create the curve c

float input = random(); //Uniformly distributed function between 0 and 1.

float scaledInput = r * (inputMax - inputMin) + inputMin;

float output = curve(scaledInput);

  The graph below shows how 10 000 random numbers are distributed. It follows the original graph closely; the discrepancy at -10 is caused by the way samples are counted (all samples between -10 (inclusive) and 0 (exclusive) are counted and plotted at -10.  

Tips and Pitfalls to Avoid

1. Generate sequences for all intermediary steps.
2. Use Excel, Calc, or some other spread sheet program to debug these sequences visually when things go wrong.
3. It is very easy to get confused with input, and output, especially after the swap. Watch out for this!
4. It is easy to get confused with the number of samples for the various sequences.
5. If you implement your own Response Curve data structure, unit tests will save you huge amounts of time.
6. Always make sure that you sample enough points, especially if your original distribution graph has rapid changes in it.
7. Always confirm that your random output follows the distribution you wanted.
8. It might be faster to use this method even when mathematical formulas are available.

Downloads

Example C++ Source Code

http://code-spot.co.za/downloads/cpp_examples/arbitrary_distribution.cpp Requires Special Numbers Library

Example Python Source Code

http://code-spot.co.za/downloads/python_examples/random_distributions.py

Related posts:

1. Generating Random Points from Arbitrary Distributions for 2D and Up
2. Generating Random Integers With Arbitrary Probabilities
3. Random Steering – 7 Components for a Toolkit