EU AI Act Risk Classification: HIGH-RISK
Annex III, Point 4(b): AI systems used to assist in decisions on promotion, compensation, task allocation, and monitoring of performance and behavior. Employment-related AI systems that influence compensation and promotion fall squarely within Annex III. The use of behavioral metadata as proxy performance indicators reinforces this classification.
NIST AI RMF Assessment
GOVERN: No documented ownership or accountability structure is described. It is unclear who is responsible for the system's outputs, who has authority to override a score, and what escalation path exists when a score is disputed. This is a blocking gap.
MAP: Foreseeable risks include disparate impact on protected classes; proxy discrimination via metadata; opacity to employees; vendor dependency on a third-party LLM; and scope creep risk.
MEASURE: No testing, bias detection, or benchmark methodology is described. For a High-Risk system under the EU AI Act, this is a blocking gap.
MANAGE: No mitigation or incident response procedure is described. Required before ship: a human oversight protocol, a dispute/correction mechanism for employees, and a vendor incident notification clause.
Privacy Intersections
Article 22 (Automated Decision-Making): If the system produces scores that managers use without meaningful independent review, this may constitute automated decision-making with legal or similarly significant effects, triggering Article 22 rights. This is the highest-priority legal question for EU/UK deployment.
Data Minimization: Email metadata and Slack message frequency are behavioral proxies. The proportionality argument for using communication volume as a performance indicator is weak without validation evidence.
Purpose Limitation: Email and Slack data were almost certainly collected for communication purposes, not performance evaluation. Repurposing for scoring requires either a compatibility assessment or fresh consent basis in the EU/UK.
Blocking Issues
Must resolve before ship: Article 22 compliance posture; Germany works council co-determination under §87(1)(6) BetrVG; legal basis for processing; third-party LLM DPA; pre-deployment bias audit methodology.
Research Flags
⚠ Whether the company's role as deployer vs. provider of a third-party LLM changes which Annex III obligations attach under the EU AI Act. Reason: emerging legislation — enforcement guidance on deployer vs. provider obligation allocation is not yet settled.
⚠ Whether a manager's use of an AI-generated performance score as a primary input constitutes "solely automated" processing under Article 22 GDPR when the manager has nominal but not substantive discretion. Reason: uncertainty — ICO and EDPB guidance does not resolve the "solely" threshold in cases of de facto managerial deference.
