How do you build a dataset that teaches the job, legally?

A dataset teaches a job only if it looks like the job. That sounds obvious until you notice how often training data is whatever was easy to collect. A model learns the patterns in its examples, whether or not they are the ones you meant.

This project builds a small dataset for one decision: route each customer message of a fictional bike shop to refunds, shipping, repairs or accounts. All 2,120 raw messages are generated by a script, so no real person's data is involved. The companion script 03_build_dataset.py needs only scikit-learn and takes seconds. If you are preparing text for a language model rather than a routing decision, this guide to preparing a dataset covers the cleaning and storage steps at larger scale.

Step 1: write the job down first

Before collecting anything, fill in this sheet. It is a shorter cousin of the datasheets researchers proposed for documenting a dataset's motivation, composition, collection process and uses.

The second-to-last row is often skipped. If two careful people disagree on the label, the model cannot do better than they can.

Step 2: run it through four gates

Each record carries where it came from. The pipeline refuses, cleans, removes duplicates and then splits, keeping a count for every step.

OK_LICENCES = {"own-records-with-consent", "cc0", "cc-by-4.0"}

def pipeline(rows):
    kept = [r for r in rows if r["licence"] in OK_LICENCES]    # gate 1: rights
    for r in kept:
        r["text"], r["scrubbed"] = scrub(r["text"])            # gate 2: personal details
    seen, out = set(), []
    for r in kept:
        key = normalize(r["text"])                             # gate 3: duplicates
        if key not in seen:
            seen.add(key)
            out.append(r)
    return out                                                  # gate 4: split by wording family

Measured here on the synthetic data:

The scrubber used patterns for email addresses, phone numbers and Canadian postal codes. It caught all 279 planted details, with no false alarms. Do not read too much into that perfect score: the script planted only the formats the patterns look for. Names in free text, such as "my neighbour Mara", pass straight through. On real data, have a person read a random sample, and treat scrubbing as a reduction of risk.

The final file ships with a manifest: row counts per label, per licence and a SHA-256 fingerprint, a short code that changes if one byte of the data changes. When a score moves next month, you can tell whether the data moved.

What the gates change, in numbers

Each figure below is an average of five runs. The model is deliberately simple: word counts fed to a logistic regression.

The first two rows use the same recipe. The difference is the split. A careless split puts near-copies of a message on both sides, so the test measures memory. Splitting by wording family, all the variations of one way of asking, gives a much more useful number. scikit-learn has group-aware splitters for this. Real equivalents of a family include one customer, one document, one recording session.

The last row is the most important in a place like Québec. An English-only dataset gave a model that scored 25% on French, the same as a coin with four sides. Accuracy on your test set says nothing about the people who are missing from it.

The legal side, in plain terms

This is general information and not legal advice. Rules differ by country and change; ask a lawyer before relying on any of this for a product.

In Canada, the Copyright Act lists fair dealing purposes such as research, private study and education, and text and data mining is not among the listed purposes. It also prohibits getting around technological protection measures, with listed exceptions. In the United States, courts are deciding these questions case by case, and how the material was obtained has mattered.

In Québec, Law 25 has strengthened the rules on personal information and the powers of the Commission d'accès à l'information. A dataset of customer messages very likely holds personal information. Using it to train a model may be a new purpose. Write down your answer, and ask the person responsible for privacy in your organization.

How do you know the data is enough before you train anything big?

Draw a learning curve: train on growing subsets and test on the same honest set. Measured here, averaged over five runs:

Past about 640 rows, more of the same stopped helping. That is a signal to change what you collect, not how much. A second experiment shows it directly. With about 140 rows, using one wording family per label gave 40.5%; the same number of rows drawn across all the families gave 83.0%. Same size, double the skill.

If the curve is still climbing at your largest size, collect more of the same kind. If it is flat and you need better results, collect what is missing: new wordings, the hard cases, the second language. Then fix label errors, because the noise table shows how costly they are, and only after that consider a larger model. Before you believe any improvement, run the honest testing in project four.