Assignment: Design a Dating App Data Safety and/or Privacy Policy

For me, the biggest tensions in designing the data schema and policies around a social application were around those of maintaining privacy while being able to enforce safety. When I was designing my application, I originally did not include identifiers such as legal name, IP address, or browser cookies that could help distinguish between users. I wanted users to stay in control of their data and to be able to delete their entire accounts.

In designing the safety policy, I realized that without identifying information it would be impossible to enforce protective account actions at the IP level. Without an IP address saved against a user account, a banned user could come back and create a new account with very little friction. It would take constant moderator vigilance to thwart the actions of a truly dedicated troll. For these reasons, I went back and added potentially identifying information such as usage data and IP address to the “data we collect” portion of the policy.

The imagined application for which the privacy policy was designed. Note here that I intended to collect very little personal information, but without identifying information, how could bad actors be effectively banned?

It seems that designing applications with privacy-minded and well-behaved users in mind paradoxically opens the door for far more avenues to exploitation and abuse than if the worst-case scenarios were considered instead. The 2023 Roberts reading this week made it abundantly clear that enforcing safety policies often takes a toll on the human moderators, too. Although AI moderation has very clear drawbacks in terms of accuracy and environmental impact, it is hard to argue that asking people to look at disturbing content for eight hours a day is much better– an NPR Fresh Air podcast paints a stark picture of content moderation work at Facebook, for example.

These privacy questions have no easy answer, and they often leave me wondering if they are “data collection problems” at all. I think back to the days of early 2000s message boards and how the relative decentralization of social spaces (and the relative lack of automation technologies) was a significant buffer against bad behavior. Trust developed more quickly in smaller communities. Content moderation was limited by the smaller spaces, and moderation decisions could be made with each community's specific interests in mind. I wonder if similar decentralization concepts applied today could help untangle the paradox of safety in data collections, such that identity could be controlled and protected while still protecting users from bad actors.

---

Resources

1. Gross, T. (2019, July 1). For Facebook Content Moderators, Traumatizing Material Is A Job Hazard. NPR. https://www.npr.org/2019/07/01/737498507/for-facebook-content-moderators-traumatizing-material-is-a-job-hazard

2. Roberts, S., Wood, S., & Eadon, Y. (2023). “We Care About the Internet; We Care About Everything” Understanding Social Media Content Moderators’ Mental Models and Support Needs. Hawaii International Conference on System Sciences. https://doi.org/10.24251/HICSS.2023.252