Microsoft patents tech to score meetings using body language, facial expressions, other data

Microsoft is facing criticism for its new “Productivity Score” technology, which can measure how much individual workers use email, chat and other digital tools. But it turns out the company has even bigger ideas for using technology to monitor workers in the interest of maximizing organizational productivity.

Newly surfaced Microsoft patent filings describe a system for deriving and predicting “overall quality scores” for meetings using data such as body language, facial expressions, room temperature, time of day, and number of people in the meeting. The system uses cameras, sensors, and software tools to determine, for example, “how much a participant contributes to a meeting vs performing other tasks (e.g., texting, checking email, browsing the Internet).”

The “meeting insight computing system” would then predict the likelihood that a group will hold a high-quality meeting. It would flag potential challenges when an organizer is setting the meeting up, and recommend alternative venues, times, or people to include in the meeting, for example.

“Because conventional computerized scheduling systems lack real-world context, users may not be aware that they are attempting to schedule non-optimal meetings, which may result in meetings that are unproductive at best,” reads a patent application made public Nov. 12. It notes, “many organizations are plagued by overly long, poorly attended, and recurring meetings that could be modified and/or avoided if more information regarding meeting quality was available.”

The approach would apply to in-person and virtual meetings, and hybrids of the two. People named in the filings include members of the company’s Azure IoT and Microsoft 365 teams. Patent filings show that the concept goes back at least two years inside the company. The new patent filing is a follow-up to a related application filed in 2018 and granted in August of this year.

The filings do not detail any potential privacy safeguards. A Microsoft spokesperson declined to comment on the patent filings in response to GeekWire’s inquiry.

To be sure, patents are not products, and there’s no sign yet that Microsoft plans to roll out this hypothetical system. Microsoft has established an internal artificial intelligence ethics office and a companywide committee to ensure that its AI products live by its principles of responsible AI, including transparency and privacy.

However, the filings are a window into the ideas floating around inside Microsoft, and they’re consistent with the direction the company is already heading.

The company rolled out its new “Productivity Score” feature in late October. The tool gives companies data to understand how workers are using and adopting different forms of technology.

Microsoft says the goal is to help organizations ensure that their workers are taking advantage of tools like shared workspaces and cloud-based file sharing to work most efficiently. This also works to Microsoft’s advantage by encouraging the use of its products such as Teams and SharePoint inside companies, making future Microsoft 365 renewals more likely.

But the tool has made headlines over the past week as reports surfaced that it lets managers see individual user data by default.

Productivity Score turns Microsoft 365 into a “full-fledged workplace surveillance tool,” wrote Wolfie Christl of the independent Cracked Labs digital research institute in Vienna, Austria. “Employers/managers can analyze employee activities at the individual level (!), for example, the number of days an employee has been sending emails, using the chat, using ‘mentions’ in emails etc.”

Announcing the general availability of Productivity Score on Oct. 29, the company anticipated and sought to address those privacy concerns. Microsoft has “a strong commitment to privacy across all our services,” wrote Jared Spataro, corporate vice president for Microsoft 365, in the post.

“Let me be clear: Productivity Score is not a work monitoring tool,” he added. “Productivity Score is about discovering new ways of working, providing your people with great collaboration and technology experiences. It focuses on actionable insights about the ways in which people and teams are using the tools so you can make improvements or provide training to further your digital transformation.”

Microsoft says it gives organizations the ability to determine who can see individual user data, with the option to anonymize or remove individual data. In addition, the post says, “to help maintain privacy and trust, the user data provided in Productivity Score is aggregated over a 28-day period.”

But critics say that’s not enough if companies, not workers, are in ultimately in control of the privacy settings.

Tying these two threads together, just imagine the reaction if the data also included information collected from cameras about body language and facial expressions in meetings, as described in the Microsoft patent filings.

By the way, one example cited in the “meeting insight computing system” patent application stood out as oddly specific, more so than any other example in the filing — striking us as perhaps an inside joke among the Microsoft team. In claims listing how the technology could be used, it cited a scenario “where the individuals in the organization are human resources managers, and one or more of the meeting trends indicate that a recurring meeting consistently has a low quality score.”

This article was written by Todd Bishop and originally appeared in GeekWire

Take Back Your Day

Learn how the latest technologies can free up your time so you can focus on your business

Share This