Social media platforms and mobile phone data are commonly mined to produce accounts of how people are responding in the aftermath of crisis events. Yet social and mobile datasets have limitations that, if not sufficiently understood and accounted for, can produce specific kinds of analytical and ethical oversights. In this paper, we analyze some of the problems that emerge from the reliance on particular forms of crisis data, and we suggest ways forward through a deeper engagement with ethical frameworks and a more critical questioning of what crisis data actually represents. In particular, the use of Twitter data and crowdsourced text messages during crisis events such as Hurricane Sandy and the Haiti Earthquake raised questions about the ways in which crisis data act as a system of knowledge. We analyze these events from ontological, epistemological, and ethical perspectives and assess the challenges of data collection, analysis and deployment. While privacy concerns are often dismissed when data is scraped from public-facing platforms such as Twitter, we suggest that the kinds of personal information shared during a crisis—often as a way to find assistance and support—present ongoing risks. We argue for a deeper integration of critical data studies into crisis research, and for researchers to acknowledge their role in shaping norms of privacy and consent in data use.