News search engine

A Search Engine Project Incorporating Query Expansion, Classification, Clustering, and Link Analysis.

• Use Boolean Search, TF-IDF, FastText, and Transformer for search
• A search engine with query expansion
• Use Naive-Bayes and fine-tuned Transformer for classification
• Use K-means for clustering
• Page Rank and HITS algorithm for link anlysis

DEMO.mov

How to start

1. install vue/cli for front project

npm install -g @vue/cli

2. install python >= 3.8 with following guide for back project
3. clone the project with following command

git clone [email protected]:IR1401-Spring-Final-Projects/News1401-13.git

4. install requirments with following command

pip install -r requirements.txt

5. some models are not on git due to maximum file size so for running project download them from links given and locate them in given path
   

• path: data/news.json | link: link
  
• path: models/classification_logistic_regression_improved/fasttext/fasttext.bin | link: link
  
• path: models/classification_transformers/pytorch_model.bin | link: link
  
• path: models/clustering/fasttext/fasttext.bin | link: link
  
• path: models/fasttext_search/fasttext.bin | link: link
  
• path: models/preprocessed_data/data.plk | link: link
  
• path: models/preprocessed_data/clf_data.plk | link: link
  
• path: models/QE_fasttext/fasttext.bin | link: link
  
• path: models/QE_fasttext/all_words_vectors_emb_fasttext.json | link: link
  
• path: models/transformers_search/transformer_model.model/pytorch_model.bin | link: link
  
• path: models/transformers_search/transformer_vectors_emb.json | link: link
  

6. run backend project with following command

uvicorn src.api:app --reload 

7. go to front/search_engine and run frontend project with following command

npm run serve

enjoy it!