TwitterDataMining

TwitterDataMining is a 2016 bachelor's thesis project that implements a complete pipeline for real-time Twitter topic detection, sentiment analysis, and visualization. It introduces WOLDA (Windowed Online LDA), a streaming topic model that uses a sliding time window to dynamically manage vocabulary and forget old topics, keeping resource usage stable. The system combines machine learning with lexicon-based sentiment analysis, achieving competitive F-scores on SemEval benchmarks (e.g., 0.714 F1 on 2013 Tweet data). Visualization includes hashtag stats, geolocation heatmaps, and interactive topic displays (treemaps, bubble charts, sunburst) to explore topic evolution and sentiment trends. Built with Python and the Twitter Streaming API, the project includes preprocessing (POS tagging with CMU ArkTweetNLP, handling slang/negation/repeated letters) and feature engineering (N-grams, sentiment lexicon scores, emoticon counts). Representative tweets per topic are selected via KL-mean, cosine distance, or maximum entropy. The code and detailed write-up are available on the author's blog. This is an academic proof-of-concept, not a maintained product. Researchers and students studying streaming topic models or Twitter analytics can use it as a reference implementation. The WOLDA algorithm's dynamic vocabulary and forgetting mechanism make it a novel approach for online LDA, but it requires adaptation for modern Twitter API changes. It is best suited for learning rather than production use.