Python, NumPy, Flask, BeautifulSoup
Sept 2024 - Present
While this was supposed to be a four-person group project, I thought I would challenge myself and program the entire search engine myself. In doing so, I taught myself the ins and outs of how search engines work: tokenization, cosine similarity, and massively optimizing the search to be efficient.
- Built a scalable search engine indexing 55,000 URLs designed for corpus growth
- Averaged <100ms latency for multi-token queries and <10ms latency for single tokens
- Implemented TF-IDF score & cosine similarity to efficiently rank and score webpages
- Efficiently handled memory usage to be less of the size of the corpus (~130MB)
