Simple Random Sampling produces samples of a specific size, where each item has the same probability of being chosen. DataFu has scalable implementations of this that will generate samples …

Sampling: simple random sample with/without replacement, weighted sample, sample by keys Hashing: SHA and MD5 Link Analysis: PageRank Assorted Macros: deduplication of tables, human-readable …

It takes a bag of n items and a sampling probability p as the inputs, and outputs a simple random sample of size exactly ceil (p*n) in a bag, with probability at least 99.99%.

Sampling Simple random sampling with or without replacement, weighted sampling. Link Analysis Run PageRank on a graph represented by a bag of nodes and edges. More Other useful methods like …

java.lang.Object org.apache.pig.EvalFunc<T> org.apache.pig.AccumulatorEvalFunc<org.apache.pig.data.DataBag> …

We can simply draw a number from this distribution, determine the positions by sampling without replacement, and then generate random scores for those positions.

It takes a sampling probability p as input and outputs a simple random sample of size exactly ceil (p*n) with probability at least 99.99%, where $n$ is the size of the population.

Sampling UDFs, including weighted sample, reservoir sampling, sampling by key, etc.

Create a new bag by performing a weighted sampling without replacement from the input bag. Sampling is biased according to a weight that is part of the inner tuples in the bag.

The method of sampling is to convert the key to a hash, derive a double value from this, and then test this against a supplied probability. The double value derived from a key is uniformly distributed …