Developing A Professor Recommendation System
Date
2024-08
Authors
Veerannagari, Vishweshwar Reddy
Major Professor
Sarkar, Soumik
Advisor
Committee Member
Gurpur, Prabhu
Sharma, Anuj
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The Professor Recommendation System is designed to connect students with professors based on the students' interests, thereby fostering academic collaboration and mentorship. This project encompasses a comprehensive approach to data collection, keyword extraction, and recommendation generation, leveraging advanced machine learning techniques.
Data collection involved scraping professor information from university websites using a Scrapy spider integrated with Selenium, as well as extracting research publication data from Google Scholar via SERPAPI. This data provided the foundation for profiling professors based on their research interests and publications. To handle the vast amount of textual data, a custom machine learning algorithm was developed, incorporating KEYBERT for keyword extraction, and further refined through phrase vectorization and count vectorization.
For the recommendation system, Sentence-BERT was employed to generate word embeddings, which were subsequently reduced in dimensionality using t-SNE. Clustering techniques, specifically K-means, were applied to group professors based on research similarities, with the optimal number of clusters determined through the Elbow method and Silhouette analysis. Recommendations were then generated using cosine similarity to match student queries with professor profiles.
A user-friendly interface was developed using Flask, HTML, and CSS, enabling students to input their interests and receive tailored professor recommendations. The system's efficacy was evaluated through feedback from 50 users, who rated the accuracy of matches as correct, partially correct, or incorrect.
Challenges encountered during the project included data inconsistencies, optimizing keyword extraction, and ensuring accurate clustering. Addressing these challenges led to several improvements and optimization strategies, including enhanced data processing techniques and algorithmic refinements.
The future scope of this system includes incorporating collaborative filtering, integrating advanced natural language understanding models, and ensuring ethical considerations in recommendation accuracy and bias mitigation. This Professor Recommendation System represents a significant step towards personalized academic mentorship, leveraging state-of-the-art machine learning to bridge the gap between student interests and professor expertise.
Series Number
Journal Issue
Is Version Of
Versions
Series
Academic or Administrative Unit
Type
creative component
Comments
Rights Statement
Attribution 3.0 United States
Copyright
2024
Funding
Keywords
Recommendation Systems,
Natural Language Processing,
Python,
Flask,
Clustering,
Word Embeddings,
Transformer Models,
BERT,
Dimensionality Reduction,
Data Analysis,
Data Engineering,
Scrapy,
Selenium,
Data Visualization,
Machine Learning,
HTML,
Statistical Analysis,
NLTK and Spacy,
ETL,
Data Pipelines,
Web Scraping