Meta Plans to Monitor Employee Mouse and Keyboard Data to Train AI

The memo reveals that the tool, called the “Model Capability Initiative” (MCI), will run on work-related applications and websites and periodically capture screenshots of employee screens. A memo released Tuesday by a company AI research scientist on Meta’s internal “SuperIntelligence Labs” channel states that MCI aims to address the weaknesses of existing AI in simulating human-computer interaction, such as selecting options from drop-down menus and using keyboard shortcuts. “All Meta employees can help our models get better through their daily work,” the memo reads.

Meta, the parent company of Facebook and Instagram, has recently heavily integrated AI into its internal workflows and is reshaping its workforce around this technology, believing it will improve company operational efficiency. According to another memo released Monday by Meta Chief Technology Officer Andrew Bosworth to employees, as part of the “AI for Work” program – now renamed the “Agent Transformation Accelerator” (ATA) – the company will strengthen internal data collection for the development of the aforementioned AI agents. Bosworth said the company’s vision is for “the future to be primarily completed by our AI agents, with our role being to direct, review, and help them improve,” with the goal of making these agents “able to automatically identify when we need to intervene, and perform better next time.”

Bosworth did not explicitly state in the memo how these agents would be trained, but he said Meta would “rigorously” collect and accumulate various interaction data and evaluation metrics involving employees in their daily work. Meta spokesperson Andy Stone confirmed that the data collected by MCI will be one of the sources for training these models. Stone emphasized that the data collected through MCI will not be used for performance evaluations or for any purpose other than model training, and that the company has implemented safeguards to protect “sensitive content,” but did not specify what types of data would be excluded from collection. “If we are to build agents that can help people with everyday tasks on computers, our models need real-world samples of human-computer interaction – such as mouse movements, button clicks, and navigating drop-down menus,” Stone said.

This practice of automating duties previously performed by human employees through AI reflects a widespread trend among large US companies, particularly in the tech industry, this year. Generative AI tools capable of performing complex tasks such as application development and organizing large amounts of data with limited human supervision have captivated Silicon Valley, while also causing significant fluctuations in the stock prices of traditional software companies and prompting some corporate executives to devise large-scale layoff plans. Meta plans to reduce its workforce by 10% globally starting May 20, and is considering implementing another round of large-scale layoffs later this year. E-commerce giant Amazon has also recently laid off 30,000 corporate employees, nearly 10% of its white-collar workforce; and fintech company Block cut nearly half of its employees in February.

Internally, Meta is urging employees to use AI agents as much as possible in tasks such as coding, even if it may reduce efficiency in the short term. The company is also weakening traditional functional distinctions between some positions, and instead introducing a universal position called “AI builder.” Last month, Meta formed a new “Applied AI” (AAI) team, aimed at improving the programming capabilities of the company’s AI models and using these models to create AI agents capable of completing most product and infrastructure development, testing, and launch work. Meta began transferring software engineers rated as “high performers” internally to the AAI team this month.

At the same time, this high-intensity monitoring plan targeting white-collar employees has also raised academic concerns about workplace privacy. Yale Law School Professor Ifeoma Ajunwa pointed out that companies previously used logging and screenshot technology primarily to find employee misconduct or work-related activities, and now using methods such as recording keyboard input for data collection further exacerbates the monitoring of employees’ real-time behavior. She said that at the federal level in the United States, there are almost no restrictions on employer monitoring of employees, and some state laws at most require employers to provide employees with a general notice when monitoring.

Valerio De Stefano, a professor of law at York University in Toronto who has long studied technology and comparative labor law, said that similar practices are likely to be illegal under European legal frameworks. In countries such as Italy, tracking employee productivity using electronic monitoring technology is expressly prohibited; and in Germany, a court previously ruled that employers may only implement keystroke logging monitoring in special circumstances, such as the existence of serious criminal suspicion. He added that, on a broader level, such monitoring practices are likely to be seen as a violation of the European Union’s General Data Protection Regulation (GDPR), and the increasing awareness of employer monitoring will also subtly strengthen the employer’s power advantage in the workplace.