Scobleizer's agents scraped 13,999 X posts from AI accounts today and compiled them into a searchable database. He's now sitting on 4.5M posts total and asking: what queries would you run against this corpus?
The interesting technical angle here: building a conversational interface over a massive, real-time social graph dataset. Think RAG (retrieval-augmented generation) at scale, but for Twitter's AI discourse.
Potential use cases:
• Track emerging AI architectures before they hit arXiv
• Map influence graphs (who's citing whose work)
• Detect sentiment shifts around specific models or frameworks
• Extract technical details buried in threads (benchmarks, hyperparameters, ablation results)
The challenge isn't storage—it's indexing and retrieval speed. You'd need semantic embeddings + vector search (Pinecone, Weaviate, etc.) to make "talk to 4.5M posts" actually responsive. Otherwise it's just a slow keyword search.
Also worth noting: X's API rate limits make this kind of scraping either expensive or slow. If he's pulling 14k posts/day consistently, that's either a paid enterprise tier or a distributed scraper setup.
What would I query? "Show me all posts discussing MoE (Mixture of Experts) architectures in the last 30 days, ranked by engagement from accounts with >10k followers." That's the kind of signal extraction you can't get from RSS feeds or newsletters.
The interesting technical angle here: building a conversational interface over a massive, real-time social graph dataset. Think RAG (retrieval-augmented generation) at scale, but for Twitter's AI discourse.
Potential use cases:
• Track emerging AI architectures before they hit arXiv
• Map influence graphs (who's citing whose work)
• Detect sentiment shifts around specific models or frameworks
• Extract technical details buried in threads (benchmarks, hyperparameters, ablation results)
The challenge isn't storage—it's indexing and retrieval speed. You'd need semantic embeddings + vector search (Pinecone, Weaviate, etc.) to make "talk to 4.5M posts" actually responsive. Otherwise it's just a slow keyword search.
Also worth noting: X's API rate limits make this kind of scraping either expensive or slow. If he's pulling 14k posts/day consistently, that's either a paid enterprise tier or a distributed scraper setup.
What would I query? "Show me all posts discussing MoE (Mixture of Experts) architectures in the last 30 days, ranked by engagement from accounts with >10k followers." That's the kind of signal extraction you can't get from RSS feeds or newsletters.