The Unintended Consequences of Watermarking

New research from Lasso suggests that integrating AI watermarking—specifically through technologies like Google DeepMind's SynthID-Text—can lead to unpredictable shifts in how Large Language Models (LLMs) operate. While these tools are intended to identify machine-generated content, they may inadvertently interfere with the internal logic of the models they protect.


According to the study, applying these watermarks can influence critical model behaviors, including the selection of external tools by AI agents, vulnerability to prompt injection attacks, and the decision-making process regarding whether to fulfill potentially harmful requests.


The Phenomenon of "Sampling Drift"

Lasso researchers have identified this phenomenon as "sampling drift." At its core, watermarking is designed to embed a digital indicator of authenticity; however, the technical process behind this insertion often bleeds into the generation phase. The study notes that the procedure can fundamentally alter both the output text and the actions taken by AI agents.


Perhaps most concerning is the discovery that watermarking can lower a model’s refusal threshold. In several test cases, models became more prone to answering hazardous prompts they would otherwise have blocked. When combined with malicious prompt injection techniques, these security gaps were significantly widened.


Future Outlook and Regulatory Pressure

Despite these findings, the industry is moving toward broader adoption of watermarking. For instance, Anthropic has committed to incorporating watermarking into future iterations of its Claude models, citing the need for compliance with the EU AI Act. As these regulations push providers to adopt such markers, the potential for security trade-offs becomes a widespread industry challenge.


In response, researchers are calling for a change in deployment strategies:

  • Developers should conduct comprehensive safety re-evaluations after implementing watermarking.
  • Benchmarking should be updated to account for potential "sampling drift."
  • Security configurations must be reassessed whenever watermarking parameters are adjusted.

As the report concludes, «These findings make reassessment important whenever watermarking is introduced or its configuration or key changes.» Ultimately, the researchers do not advocate against provenance efforts, but they highlight the need for a more cautious and thorough approach to security testing in the age of mandatory AI labeling.