By Isha Suri and Shashank Mohan

A recent investigation by the BBC Eye in India found that Instagram was selling advertisements that contained child sexual abuse imagery (CSAM) and providing users with links to buy CSAM material for as little as Rs 99. The only formal action from the Indian government is a show-cause notice to Meta (parent company of Instagram), including asking it to take down the abusive content. Initially, when flagged by the journalist, Instagram stated that the said ads did not violate its “community guidelines”, however, later a company spokesperson was quick to clarify that Meta has “a zero tolerance policy” for soliciting or sharing CSAM including in advertisements. As expected, the policy response focused on “content moderation” without acknowledging the perverse economic incentives underpinning the advertising-led business models of Big Tech companies including Meta.

Social media platforms operate in multisided markets as intermediaries for distinct user groups, where one-side of the market is “zero-priced” with monetisation from the other side compensating for this apparent subsidy. For instance, on one side, social media companies enable users to generate and upload their content for ‘free’, but on the other side they sell personalised placement slots to digital advertisers for revenue generation. According to estimates, digital advertising accounts for nearly 97% of Meta’s and nearly 79% of Google’s annual revenue. These advertising-led business models dependent on massive data collection, profiling, and personalisation is the major source of revenue for social media platforms.

Needless to say, user engagement remains the most important aspect of ensuring profit maximisation through advertising. Research suggests that toxic and fabricated content is likely to be more engaging, with one study reporting that false news was  likely to spread six times faster than the truth. Consequently, algorithms are designed to maximise user engagement, including extreme content and issues that contribute to the formation of filter bubbles seeking to reinforce existing beliefs. An internal study by Facebook revealed that its News Feed algorithms exploit the human brain’s attraction to divisiveness and if left unchecked it would feed users “more and more divisive content in an effort to gain user attention and increase time over platform”. Similarly, employees at Google sought to improve issues pertaining to filter bubbles and enhance diversity of content by modifying YouTube’s recommendation algorithm. However, it reduced engagement and the change was ultimately rolled-back. Owing to their integrated structures and profit maximising incentives inextricably tied to “views”, platforms continue to employ algorithms that recommend divisive and harmful content with absolute disregard for consumer safety or wellbeing. At the very least, we must ask ourselves when revenue is tied to user attention, what incentives exist to prevent harmful content from being amplified? 

Amidst all the cacophony, a questionable interpretation of the Indian safe-harbour protection has emerged. The safe-harbour law protects intermediaries (think social media, search engines, cloud services, but also telecom networks) from third-party content carried by them, essentially providing them immunity from content liability. BBC’s investigation suggests that social media platforms such as Instagram may escape prosecution potentially due to the safe-harbour rule, which might lead to a renewed call for “stricter content moderation laws” or dilution of the safe harbour principle in its entirety. However, this is an erroneous reading of the safe-harbour law. As such, Instagram ads cannot be equated to independent user-generated posts or third-party content. As a commercial transaction, ads get displayed once the content has been reviewed. Hence, the rationale behind safe-harbour doesn’t apply to ad-related content. Social media platforms including Meta cannot hide behind inaccurate content detection algorithms for ads or otherwise, particularly for egregious harms from content like CSAM. This is also reflective of the complex and arbitrary nature of India’s digital media regulatory ecosystem, wherein content moderation facilitated through a provision of a nearly three-decade old law is treated as a silver bullet and frequently deployed to regulate all digital services. Over-reliance on intermediary liability fails to mitigate modern-day online harms, and unless the underlying structural imbalances between dominant corporations, the state, and citizens are addressed, digital freedom and safety in India will continue to remain elusive.

The lure of safe-harbour

This safe-harbour law has historically facilitated the development of the online ‘public square’. The underlying principle is straightforward: without fear of liability, online platforms could enable unfettered information and communication channels for variegated content over the internet. Despite its shortcomings, particularly with respect to arbitrary crackdown on online speech, inconsistent moderation, and opaque algorithms, these platforms are an essential public utility, as demonstrated by recent youth-led protests that mobilized online support across India and tried to hold authorities accountable to higher standards. However, over time, safe-harbour protection in India has become conditional upon compliance obligations drawn exclusively by the central government under delegated legislation. These rules, contained in the Intermediary Guidelines and Digital Media Ethics Code Rules, 2021 (IT Rules), originally intended to provide content moderation guidance to internet intermediaries, are now being used to regulate a host of digital services including, but not limited to, news publishers, streaming platforms, AI chatbots, online games, and even texting apps to enable traceability of the first originator of a message! In 2023, the government proposed establishing a “Fact Checking Unit” to monitor online content related to “any business of the Central Government”, and order takedown of content it deemed fake or misleading. Failure to comply would invite liability for platforms hosting such content.  Although, the proposed amendment was struck down as unconstitutional by the Bombay High Court, this illustrates how the threat of withdrawing safe-harbour protection can be leveraged to  expand State control over social media platforms, compelling companies to over-correct (read take down harmless and lawful content), ultimately resulting in a chilling effect on online speech. This also enables the State to operate as the arbiter of truth with grave consequences for freedom of speech and expression. 

More importantly, the  IT Rules have expanded far beyond their original rationale leading to a lack of regulatory coherence. With an ever expanding remit, ranging from  traceability to rules on online gaming, and most recently generative AI content, it has become the government’s default instrument for digital regulation. This continued expansion through delegated legislation raises serious questions around democratic accountability and constitutional limits on executive rule making. 

Despite the overzealous regulatory efforts put behind frequent amendments to the IT Rules, sole reliance on the safe-harbour regime is unlikely to address contemporary digital harms and fails to check abusive conduct by dominant technology platforms. For instance, if the government takes a social media platform to court for not taking down non-consensual intimate imagery (NCII), posted by a third-party upon receiving notice (IT Rules, r. 3(2)(b)), the platform can simply remove the offending content at a later stage, and in doing so escape liability, even though the harm has already occurred. Platform liability only arises when they knowingly carry illegal content as per the safe-harbour law. Similarly, a teenager could use a Large Language Model (LLM) to generate a morphed image of a classmate and post it online without their consent (IT Rules, r. 3(3)) (by design, the IT Rules aren’t meant to regulate AI chatbots, that currently don’t allow for third-party interaction, AI governance guidelines from the government, recognises this is a challenge too). The company operating the LLM could subsequently tweak their model and argue that it no longer allows image morphing. However, not only does such post hoc compliance fail to prevent the initial harm, it also allows dominant platforms to evade accountability, while simultaneously enabling the government to coerce companies into complying with arbitrary and unconstitutional content takedown orders. 

The proverbial elephant in the room

Today, a handful of technology giants wield immense power over the digital sphere by gatekeeping essential services that people and businesses depend upon to access information and exercise their fundamental right to free speech and expression. Most policy proposals have solely focused on what is “allowed” or “removed” from these platforms. These measures may address individual instances of harm, but effectively fail to mitigate the structural incentives and market power that enable the rise and amplification of such harms.

Meanwhile, existing competition and antitrust frameworks provide far more robust tools to address the underlying structural issues and economic incentives of the dominant social media platforms. However, these have thus far remained on the fringes of policy discussions. These interventions could include behavioural remedies such as requiring platforms with significant market power to functionally separate content curation and hosting services from advertising services, or structural remedies such as break-ups, divestitures, unwinding completed mergers, and asset transfers. Competition authorities may also subject future mergers and acquisitions by dominant platforms to stricter scrutiny to prevent further concentration and ensure plurality and diversity of choices available to consumers. Notably, Section 28 of the Competition Act, 2002 empowers the Competition Commission of India (CCI) to divide an enterprise to prevent abuse of dominance. In this particular instance, competition law can therefore play an important role in reshaping the market incentives that shape what platforms choose to amplify. However, designing effective structural and behavioural remedies requires competition authorities to develop substantial expertise in the digital sector and must be done on a case by case basis.

It is equally important to acknowledge that content moderation rules including the intermediary guidelines cannot compel platforms to make their services less addictive, provide meaningful  alternatives to opaque algorithmic feeds, prohibit addictive features like infinite scrolling, mandate  interoperability; or implement  transparency measures such as  algorithmic audits and risk assessments– measures that other jurisdictions have implemented. As LLMs become increasingly embedded into modern human lives, expecting the current safe harbour regime to  govern AI systems that prioritise factual accuracy over reinforcing misconceptions and encouraging risky behaviour including self-harm stretches the doctrine well beyond its original purpose. The Indian government continues to vacillate on the issue of AI regulation. They have used advisories and IT Rules, both under the IT Act, to regulate the generation of harmful synthetic content, with recent reports also indicating that the government is  deliberating on a new AI law. This is a departure from its previous position that existing regulations including the IT Act, Digital Personal Data Protection Act, and intermediary rules are sufficient to address AI related harms. Such regulatory uncertainty and piecemeal approach creates uncertainty while failing to address the structural incentives underpinning online harms. More importantly, content takedown measures will not alter the underlying market structures and economic incentives rewarding the amplification of harmful content – an issue that competition law is uniquely placed to address. To build a genuinely safe and inclusive digital ecosystem, we must reimagine our approach to digital media regulation and move beyond using intermediary liability as a silver bullet to check every digital harm. It is long overdue to move beyond the safe harbour regime as the only tool for platform accountability. The Indian experience demonstrates that not only is it a blunt instrument while ascribing liability but it also empowers the Executive to censor online speech, with disproportionate consequences for marginalized communities, while doing little to hold platforms accountable. Building a safer digital ecosystem requires looking beyond the safe harbour to a more polycentric model of governance comprising competition policy, transparency obligations, independent audits and due process safeguards to hold both State and platforms accountable, while foregrounding citizen interest. 

Isha is an independent researcher and lawyer based out of New Delhi, India. She was also the 2025 Global AI and Market Power Fellow with the European AI Society Fund. 

Shashank is a New Delhi-based lawyer focused on technology law and policy research.

Read more