California nonprofit corporation · charitable/tax-exempt recognition pending
Institute for AI Ethics in Human Cognition
Advancing AI safety by keeping humans in the loop where cognition, consent, and connection are at risk.
We are a newly formed, independent nonprofit working at the intersection of artificial intelligence and the most intimate parts of human life — memory, judgment, care, and companionship. Our work is education, research, and the development of public-interest standards, with particular attention to people whose cognition makes them more vulnerable to automated influence.
01 — Purpose
Why this Institute exists
The Institute for AI Ethics in Human Cognition advances education, research, and public-interest standards for the safe and ethical use of artificial intelligence where cognition, consent, and human connection are at risk, with particular attention to cognitively vulnerable populations and human-in-the-loop safeguards. Formal purpose statement
In plainer language: we study what happens when AI systems enter the places where people think, remember, decide, and depend on one another — and we work to make sure a person remains meaningfully in charge.
Our attention is on the situations where the stakes are highest and the protections are thinnest: dementia care and cognitive decline, mental health and isolation, caregiving relationships, and any setting where a person may not fully recognize that they are speaking with a machine, or may not be positioned to refuse.
02 — Why now
AI has moved into intimate domains
For most of its history, artificial intelligence sat at a distance from daily life — in search results, logistics, and back-office decisions. That distance has closed. Conversational systems now sit inside cognition, care, companionship, consent, and connection: reminding someone to take medication, keeping a lonely person company at midnight, summarizing a diagnosis, or standing in for a relationship.
These are not ordinary product settings. They involve people who may be tired, grieving, isolated, cognitively impaired, or simply trusting. Familiar consumer safeguards — a terms-of-service checkbox, a disclaimer, an opt-out buried in settings — were not designed for a person with memory loss, or for a caregiver making decisions on someone else's behalf.
We believe this gap is addressable, and that it is better addressed early: with evidence, with practitioners and families in the room, and with standards that assume a human being remains accountable for what happens.
- Capability is arriving faster than practice. Deployment in care settings is outpacing the guidance available to the people responsible for it.
- Consent is doing more work than it can bear. Fluctuating capacity, surrogate decision-making, and always-on systems strain conventional consent models.
- Attachment is a design outcome. Systems that feel like relationships create dependency risks that must be anticipated, not discovered.
03 — The standard
Presence is not oversight
Nearly every framework for medical AI now contains the phrase "human in the loop." We agree with the instinct, and we think the phrase, left undefined, is dangerously incomplete. A clinician who signs off on two hundred automated recommendations an hour is in the loop in the trivial sense; he is not overseeing anything. Worse, he absorbs responsibility for failures he could not realistically have caught — what the anthropologist Madeleine Clare Elish has called a moral crumple zone: the human component that takes the impact when a complex automated system fails.
So the question worth answering is not whether a person should remain in the loop; almost everyone says yes. It is what has to be true for that person's presence to mean something. We propose four conditions. Oversight that omits any one of them is, in our reading, oversight in name only.
-
Protected time
A floor on review minutes per decision and a ceiling on caseload. Supervision that fails arithmetic is not supervision; if the schedule makes genuine review impossible, the oversight was theater before anyone made a mistake.
-
Logged override authority
The human can countermand the system, and the system records both when they do and when they do not. Authority that leaves no trace cannot be audited, and authority that is never exercised deserves examination rather than trust.
-
Retained liability
The person keeps professional responsibility for the decision. This is the load-bearing clause: it is the one safeguard that cannot be quietly priced away, and the most reliable reason a busy person keeps actually looking.
-
Information sufficiency
The system surfaces its uncertainty and the evidence against its own answer, not a single confident output. A person handed only a conclusion is ratifying; a person shown the doubt is deciding. The distinction is the whole point.
The concept of the moral crumple zone is drawn from Madeleine Clare Elish, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (Engaging Science, Technology, and Society, 2019). A white paper formalizing the standard and its scientific basis is in preparation.
04 — In practice
What we ask of AI that touches patient care
Principles earn their keep when they become requirements someone can check. Where an AI system participates in the care of a patient — above all a cognitively vulnerable one — we hold the following to be minimums rather than aspirations. They begin from a single premise: responsibility for a patient rests with an accountable human being — a physician, a nurse, a family caregiver — and never with the machine.
- Non-deceptionThe system may not claim to be a person, a family member, or a clinician, and may not exploit misrecognition or identity confusion. Familiar-presence cues are permissible only without false first-person or relational claims.
- A human accountable at every stageCaregiver- or clinician-in-the-loop by default. Sessions, memory, and delivery remain under a named person's control; the system holds no persistent goals of its own.
- Clinical boundariesNo diagnosis, no dosing, no medication changes, no emergency guidance. Urgent symptoms and risk are escalated to humans, not managed by the machine.
- Escalation to peopleSelf-harm, abuse, exploitation, and crisis content trigger immediate human hand-off, under defined review triggers rather than discretionary ones.
- Safety over engagementThe system is built to redirect, refuse, or end a session rather than maximize attention. Dependency and parasocial attachment are treated as failure modes, not growth metrics.
- AuditabilitySafety triggers, moderation actions, and rule changes are logged and reviewable, and behavior is verified against a versioned test suite rather than taken on faith.
- Privacy minimizationCollect the minimum, disclose what is stored and who can see it, and delete on a schedule. Intimate disclosures are not a data asset.
- Transparency of limitsPlain-language disclosure that it is a machine, what it cannot do, and that it may be wrong — written for the person actually using it.
A standard is a claim until something is built and measured against it. The Institute's first applied program is CARE-SAT, a small feasibility study of a safety-moderated companion for dementia caregivers, designed to test whether these constraints survive contact with daily use. The tool under study, Lucid Bridge, is developed by a separate commercial entity; the Institute's role is the standard and the evidence, and its conclusions do not depend on the product's success.
05 — Values
The commitments that govern our work
Nine principles guide us — human oversight, dignity, consent, safety, transparency, accountability, evidence, independence, and public benefit. They gather into six working commitments.
-
i
Human oversight
A person — not a system — holds responsibility for consequential decisions. Automation may inform and assist; it should not quietly replace judgment.
-
ii
Dignity
People are not problems to be managed. Tools used in care should preserve autonomy, identity, and the right to be treated as a full participant in one's own life.
-
iii
Consent
Consent must be meaningful, revisitable, and appropriate to capacity — including where a caregiver or surrogate is involved. Silence is not agreement.
-
iv
Safety & transparency
People should be able to tell what they are interacting with, what it can and cannot do, and where its limits lie. Failure modes should be anticipated and disclosed.
-
v
Accountability & evidence
Claims should be testable and traceable. We aim to publish our reasoning and methods, and to revise our positions when the evidence warrants it.
-
vi
Independence & public benefit
Our conclusions are not for sale. We intend to organize our work, funding, and disclosures so that the public interest remains the deciding consideration.
06 — Goals
What we intend to build
The Institute is newly formed. The following describes intended directions rather than completed programs; scope and sequence will depend on capacity, partnerships, and funding.
-
Education
Plain-language materials and briefings for families, caregivers, clinicians, and practitioners on what AI systems in care settings can and cannot responsibly do.
-
Research
Studies and structured reviews of how AI-mediated interaction affects cognition, dependency, consent, and wellbeing — conducted with appropriate ethical review and collaborators.
-
Public-interest standards
Draft guidance, evaluation criteria, and disclosure practices developed openly, for voluntary adoption by developers, care organizations, and procurement teams.
-
Caregiver & vulnerable-population safeguards
Practical protections for people with cognitive impairment and for those who support them, including guidance on capacity, surrogate consent, and escalation to a human.
-
Human-in-the-loop design guidance
Concrete patterns for keeping people meaningfully in control: oversight checkpoints, refusal behavior, honest self-description, and hand-off to accountable humans.
-
Pilot work on ethically constrained care tools
Exploratory, small-scale pilots examining whether AI-assisted care tools can be built within strict ethical constraints — undertaken cautiously, with oversight, and reported honestly whether or not results are favorable.
07 — Initial focus areas
Where we are starting
- Cognitively vulnerable populationsDementia, cognitive decline, and other conditions that affect how a person evaluates, remembers, or resists automated influence.
- AI-mediated careSystems that participate in caregiving — reminders, monitoring, triage, companionship, and support for family and professional caregivers.
- Parasocial & dependency risksAttachment to systems that simulate relationship, and the effects of that attachment on isolation, trust, and decision-making.
- Human-in-the-loop safeguardsDesign and governance patterns that keep an accountable person present at the points where it matters most.
- Safety standardsEvaluation, disclosure, and deployment criteria for AI used in cognitively sensitive and care-adjacent contexts.
08 — Contact
Stay informed
The Institute is in its earliest stage. We expect to share published materials, draft standards, and collaboration opportunities as the work develops. We are particularly interested in hearing from clinicians, caregivers, researchers, ethicists, and engineers working on these questions.
Contact information coming soon.
This page is an announcement of the Institute's formation. We are not soliciting contributions at this time.