Strengthen ranking and visibility controls against low-reputation accounts distributing illegal content (tags, real-time, engagement, media)

[segior340-source]

Summary

In the X platform/app illegal content distributors currently exploit several weaknesses in the ranking and visibility system by using real-time search. Bad actors rely heavily on tags and repeated text patterns; when one tag is blocked they simply change a letter and continue the cycle. Low-reputation accounts receive the same real-time ranking as established ones, allowing multi-account operators to keep illegal material continuously visible in certain tags. Also illegal posts from recent accounts often remain visible for about a minute, which is enough time for rotation across multiple accounts.

Description

Analysis of the publicly available recommendation and visibility components (user-mass/tweepcred, visibilitylib, trust-and-safety models, graph signals, entity/hashtag handling, and ranking pipelines) reveals several gaps that significantly lower the cost of operating account farms for illegal content:

1. Illegal content distributor accounts rely on text and tags to distribute their content. When one tag is blocked they change a letter and the cycle repeats.
2. Low-reputation accounts currently receive the same ranking in real-time search as established accounts, allowing multi-account operators to maintain continuous visibility of illegal content in certain tags.
3. In real-time posts it is possible to see obvious illegal content from recent accounts if the post is not older than ~1 minute.
4. Engagement (likes, reposts, replies) with content that later receives strong abuse/illegal safety labels does not reliably propagate a negative signal back to the engaging accounts. Many of these accounts operate in “incubation mode” and are later used to distribute illegal content.
5. Following or heavily engaging with accounts later banned for illegal content currently triggers little or no automatic soft penalty or warning.
6. Text + tag combinations previously associated with banned accounts are not strongly penalized via embedding or fingerprint similarity when reused.

These weaknesses exist even though the necessary building blocks are already present in the open-sourced code. Closing them would raise the operational cost of distributing illegal content at scale without requiring disclosure of closed adversarial models.

Suggestions to increase safety and make bad actors’ work harder

1. Turn high-risk tags into a “killer zone”
   Bad actors routinely use tags (including simple letter variants such as #teenageer) to give new or aged accounts visibility. When a tag is banned they change a letter and continue. Tags that appear frequently on banned illegal-content accounts should carry a strong negative score. Any account that uses them should lose significant visibility, even if it has not yet posted illegal material, because it is highly probable to be a hidden distributor. This makes the tag itself expensive and risky to use for distribution.
   Note: Bad actors could try to attach these tags to legitimate content to poison it, but doing so would reveal their intentions and is also very costly for them.

2. Penalize engagement with banned content
   Accounts that like, share, or repeatedly engage with content that is later banned for illegal material should receive a reputation or ranking penalty. Many of these operations rely on “incubation” accounts that look normal and exist primarily to boost the bad content (and later distribute it themselves). Making that engagement costly breaks the boosting loop.

3. Restrict real-time ranking/distribution to good-behavior accounts
   Real-time posts currently give spammers a major advantage. Even when obvious illegal content is banned in about one minute, operators with multiple accounts can keep material continuously visible in tags. Full real-time ranking and distribution should be reserved for accounts with established good behavior. Low-reputation or new accounts should face delayed or reduced real-time visibility.

4. Add a short delay for images and videos from low-trust accounts
   Media posts should not become fully visible immediately when they come from low-trust accounts. A brief hold to run a quick illegal-content check (using existing media classifiers such as pNSFWMedia) would close the short window that multi-account operators currently exploit.

5. Automatic warnings for accounts that follow banned illegal-content accounts
   Accounts that follow profiles later banned for illegal content should receive automatic messages warning about the association. Persistent patterns can trigger further review or soft penalties. This creates an early signal without requiring immediate full investigation of every case.

Impact

Implementing these measures would make large-scale distribution of illegal content more expensive and less effective, reduce the window during which such material stays visible, and disrupt the boosting and incubation tactics currently used by bad actors, while having minimal impact on legitimate high-reputation accounts.

Supporting Material / References

• Open-source repository: https://github.com/twitter/the-algorithm
• Key components referenced:
  • UserMass.scala / tweepcred
  • visibilitylib (including SensitiveMedia and safety-label rules)
  • trust_and_safety_models (pNSFWMedia, abuse, toxicity)
  • real-graph
  • SimClusters / TwHIN embeddings
  • Topic Social Proof
  • Entity extraction (hashtags)
  • home-mixer ranking pipelines

Related public discussion of ranking and visibility behavior is available in the repository issues and the engineering blog post linked from the README.