Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jatin get profile images from website #1116

Open
wants to merge 10 commits into
base: development
Choose a base branch
from

Conversation

JatinAgrawal94
Copy link
Contributor

@JatinAgrawal94 JatinAgrawal94 commented Sep 22, 2024

Description

Screenshot 2024-09-08 at 1 41 52 PM

Related PRS (if any):

To test this backend PR you need to checkout the #2714 frontend PR.

Main changes explained:

  • Functions added in usercontroller.js for removing and updating profile images.
  • userhelper function created in userhelper.js for getting images from website.
  • New key added to userProfile Collection names suggestedProfilePics to store images from website.

How to test:

  1. check into current branch
  2. do npm install and ... to run this PR locally
  3. Go to server.js and enter the line of code shown in the video as "for testing purposes only",
  4. Make changes to the query as stated in the video.
  5. Run npm run build and npm start .
  6. Go to frontend repo for next steps.

Screenshots or videos of changes:

profile_images.mp4

@Ankuriboh
Copy link
Contributor

Code work great, but I don't think it is necessary to introduce an extra dependency (puppeteer). Since we only care about the simple content on the webpage, use just fetch to scrap the page and cheerio to parse the data is already sufficient enough.

Asides from the code part, I wasn't quite sure why this functionality was needed since we already had the functionality to set avatar in the app. I've commented on the bugs docs for the same issue.

package.json Show resolved Hide resolved
src/helpers/userHelper.js Outdated Show resolved Hide resolved
src/helpers/userHelper.js Outdated Show resolved Hide resolved
src/routes/userProfileRouter.js Show resolved Hide resolved
src/routes/userProfileRouter.js Show resolved Hide resolved
Copy link
Contributor

@samarth9201 samarth9201 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the PR and provided my comments in Frontend PR. The code looks okay, but I am facing issues as mentioned in Frontend PR.

package.json Outdated Show resolved Hide resolved
@JatinAgrawal94
Copy link
Contributor Author

Code work great, but I don't think it is necessary to introduce an extra dependency (puppeteer). Since we only care about the simple content on the webpage, use just fetch to scrap the page and cheerio to parse the data is already sufficient enough.

Asides from the code part, I wasn't quite sure why this functionality was needed since we already had the functionality to set avatar in the app. I've commented on the bugs docs for the same issue.

Removed puppeteer, using fetch to reduce load on the server. Please review again.

@Hritvik111 Hritvik111 added the High Priority - Please Review First This is an important PR we'd like to get merged as soon as possible label Nov 28, 2024
@JatinAgrawal94 JatinAgrawal94 dismissed Ankuriboh’s stale review January 6, 2025 02:45

fixed the changes.

@JatinAgrawal94 JatinAgrawal94 force-pushed the jatin_get_profile_images_from_website branch from dd0c2db to 840e7b9 Compare January 6, 2025 04:03
Copy link

@honglinchen0524 honglinchen0524 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work Jatin! Here are some of my suggested changes.

const getProfileImagesFromWebsite= async () => {
try {
// Fetch the webpage
const response = await axios.get("https://www.onecommunityglobal.org/team");

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding timeout and retry logic. Without timeout and retry logic, if the network requests fail or hang, it could cause the application to become unresponsive.

return false;
}

async function imageUrlToPngBase64(url) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding image size limit or image compression to ensure large size images won't slow down the server significantly.

});
var users=await userProfile.find({'isActive':true},"firstName lastName email profilePic suggestedProfilePics")

users.map(async(u)=>{

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your current implementation using map with async callbacks. I think this might lead to race conditions, since it doesn't wait for all promises to resolve. I would suggest using Promise.all to properly handle multiple asynchronous operations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
High Priority - Please Review First This is an important PR we'd like to get merged as soon as possible
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants