[Through BCL India] An Inquiry — Part 2

You can view the full report here: link.

Excerpts:

At the level of a single human being — not a labour union, not a climate activist group, not a group of federal bankers — AI is no longer the magic wand that grows from strength to strength (or from one Claude announcement to another). It comes with legitimate concerns — labour displacement, hallucination and misinformation, sycophancy, behavioural manipulation, and an erosion of mental faculties, among others.

[…]

In an epoch of big data, that downside looks like what Professor Shoshana Zuboff called ‘surveillance capitalism’ - a practice that “unilaterally claims human experience as free raw material for translation into behavioral data.” At some level, advocating for data privacy at this stage almost feels like a lost cause. After all, ‘companies now know more about our personal lives than we ever thought they would: who we are, what we like, where we go, what we do, whom we do it with, and what we think and even feel.’ (Stanford AI Index Report, 2026.)

But consumers and regulators must reckon with it nonetheless. The lived human experience cannot become a mere data point that’s employed to develop technology that we’re still figuring out the boundaries of. Not anymore than it already has anyway.

[…]

There’s no ‘us’! There’s you, a living, breathing human on this side of the screen, and a ‘giant statistical prediction machine’ (IBM, ‘What are large language models’) the other side (or more precisely, somewhere in a data centre). But with the increasing enmeshment into everyday choices (what do I eat, where do I travel to, is it me or my friend in the wrong) and the simultaneous advancement of autonomous AI agents, we are at the risk of losing the ability to notperceive Claude or ChatGPT as a helpful, well-wishing, generous friend. And that means that we are officially in dangerous territory with the launch of advertisements in ChatGPT.

[…]

Optimising the act of communicating ‘look, here’s a fantastic product!’ for productivity and efficiency gains is all well and good, but it also erodes human intent from the act — and in the absence of human intent on one side of the equation, it is but inevitable that human intent on the other side is also found lacking.