Show Us the Data: Privacy, Explainability, and Why the Law Can’t Have Both*

Two rights—the right to privacy and the right to an explanation when automated decision-making affects an individual—are in fundamental conflict. To recognize the conflict, one needs to understand two things: (1) how machine learning works and (2) how a litigant in a dispute against a data controller will use litigation procedure to demand both explanation and proof of the validity of any explanation proffered.

Machine learning is not based on logic-driven algorithms explicable by examining source code. It is a data-driven process of pattern-matching based on large data sets and statistics. Data subjects affected by machine learning decisions will demand different explanations depending on each subject’s relationship to the data controller. Where the data controller and data subject trust one another, the explanation of the decision might be given in general or “accessible” terms. But where they are in conflict and the subject has resorted to adversary procedures, the dynamics of litigation will impel the subject-claim- ant to demand not only an explanation, but proof that the explanation is accurate and true. This process of demand leads to an intensifying scrutiny of the data that trained the machine. It inevitably will lead to demands to see the training data—not anonymized or partial versions of it—and thus will trespass upon the privacy rights of people from whom the training data was derived.

Various solutions might be attempted, such as data rooms or in camera review. There are reasons to be skeptical of these and, in any event, they remain to be tested. The fundamental conflict, regardless of compliance approaches adopted to mitigate regulatory risk, seems likely to be exposed and played out chiefly in adversarial proceedings in the years ahead.

The Article takes as its focus the European Union’s General Data Protection Regulation because of this regulation’s global prominence, but the difficulty it identifies will be presented across national jurisdictions, as many legislatures adopt explainability laws against the backdrop of long-vested privacy rights.