Pagination

[DevinCLane]

Moving the conversation from #7 to here while we iron things out @MainlyColors @Caleb-Cohen @timmyichen

Summary:
This tech spec outlines the technical requirements for implementing pagination on Dev Directory

Scope:

• Displaying 20 profiles to start
• Implementing infinite scroll

Data model:

No changes needed to the data model. This tech spec is only responsible for reading from the DB.

User Stories:

1. User lands on pagination page with a list of 20 developers within Dev directory.
2. More profiles are shown via infinite scroll when the user scrolls down.

Implementation:

• Use Sequelize for queries, e.g.,

  const users = await User.findAll({
    attributes: ['id', ...otherFields],
    limit: 20,
    raw: true,
  });

• We will paginate based on [Offset API Pagination] (https://nordicapis.com/understanding-5-types-of-web-api-pagination/#offset-api-pagination)

• More information on pagination approaches

• Infinite scroll

  • use IntersectionObserver to detect when the user has scrolled to the bottom of the page
    • Using intersection observer
  • Re-render on filter decisions. For example:

  useEffect(() => {
    if (!cardRef?.current) return;
  
    const observer = new IntersectionObserver(([entry]) => {
      if (isLast && entry.isIntersecting) {
        newLimit();
        observer.unobserve(entry.target);
      }
    });
  
    observer.observe(cardRef.current);
  }, [isLast]);

APIs:

• Use a single endpoint: GET /api/users. Using query parameters to control what is returned.

Security:

N/A

Testing:

• Upon initial load, 20 profiles are shown to the user.
• Pagination feature is tested with different amounts of data to ensure it performs well with large and small data sets.

Open Questions:
N/A

Deployment:

N/A

[Caleb-Cohen]

Reposting my question:

@DevinCLane I want to walk back the randomness. I think this will add complexity and solve a problem we won't even have with MVP due to limited users.

I'd recommend lowering the complexity and setting pagination to just a sort of some default criteria.

[MainlyColors]

(copying same comment from issue #7)
For the random implementation (line below) what guarantees on page 2 that a user from page 1 isn't selected again when randomly selecting from the DB?

const res = await client.query('SELECT * FROM users ORDER BY random() LIMIT 8');

[devonzara]

Apologies for not being able to attend the initial meeting (and for the length of this post). After reviewing the document, I have a few points that I would like to discuss:

Name/Summary:

• This tech spec outlines the technical requirements for implementing pagination on Dev Directory

  The name/summary states "Pagination", but the contents seem to cover search and filters too.
  Might be good to create a separate spec for the search/filters and keep this limited to building the initial directory view(s) and pagination feature.

Data Model:

• Users

  • individual users
  • profile picture
  • bio
  • mentor or mentee status
  • ...

  No changes needed to the data model. This tech spec is only responsible for reading from the DB.

  It states no schema changes are necessary, but also mentions user fields that don't exist.
  Additionally, I'd say worrying about what is displayed is also outside the scope of "Pagination."

User Stories:

• As the user filters, profiles are removed/added to the page, results should be randomized to provide diversity.

  Filters may fall outside of scope.

• Results should be randomized such that multiple requests for results with same filters produce different results. However, if there are more results than viewable slots, it should not be randomized on every addition of a filter. For example, if half of the currently rendered profiles fit with the next applied filter they should not be removed because of randomness.

  Following a discussion with @Caleb-Cohen, I believe we agreed that we should exclude randomness for now. Additionally, it's important to note that randomization may hinder shareability.

Implementation:

• Not sure the need for such an extensive code example unless we need to explain a very specific approach. In this case, the project makes use of Sequalize; so we'll rarely need to write raw SQL queries. Instead, we'll likely be using something like:

  const users = await User.findAll({
    attributes: ['id', ...otherFields],
    limit: 8,
    raw: true,
  });

• We will paginate based on Keyset API Pagination. For example:

  Can you share your thoughts on why Keyset API Pagination was chosen over Offset API Pagination?

APIs:

• We should only need a single endpoint, GET /api/users... using query parameters, we can control what is returned.

  Pending the results of our meeting tonight, we may use this or NextJS' built-in getServerSideProps() function to handle such requests. We've decided to go with the RESTful approach.

Testing:

• Different devices and screen sizes: does this behavior work across screen sizes and devices such as mobile, tablets, etc. Responsive design.

  We don't currently have a design, therefore I wouldn't worry about responsiveness.

Open Questions:

• Is the Search feature a part of this pagination feature? Search might need a separate tech spec.

  I'd say "No", Search/Filters probably shouldn't be a part of this spec and should instead be it's own.

• Is Pagination accessible to users who are not authenticated? Will profiles be displayed automatically? What restrictions are in place for users who are not auth'ed?

  I do believe Guests should be able to browse beyond the first page.

[DevinCLane]

Can you share your thoughts on why Keyset API Pagination was chosen over Offset API Pagination?

No strong preference either way, just went with what looked quickest to MVP

[DevinCLane]

Following a discussion with @Caleb-Cohen, I believe we agreed that we should exclude randomness for now. Additionally, it's important to note that randomization may hinder shareability.

Sounds good to me, we can deal with the need for randomness later

[DevinCLane]

Overall @devonzara I am in favor of these comments and feedback.

[timmyichen]

👍 general process during the review is - the original post should be edited to incorporate any feedback received. then check in with folks to make sure it looks good to them, then you're free to make tickets! i've added a label for the pagination project so just tag newly created issues with that tag

[DevinCLane]

ok!

[DevinCLane]

Updated things a bunch, would love another look at this @timmyichen @devonzara @Caleb-Cohen @MatthewBozin etc

[DevinCLane]

GET /api/users/:id should not return the discord_name if the caller is not authed

We need to display something, if not the discord_name... what? I think we need to display them, at least until we decide on something else.

just deleted the Security section

[DevinCLane]

Looking good! A few comments:

* We probably want to have a page size larger than 8 to ensure that we _can_ scroll if we're doing infinite scrolling (see second bulletpoint).  In any pagination is a way to reduce information into pieces that are consumable by users and won't overload the database/network, and I think both our brains and our computers can handle more - I'd say 20-25 :)

* We should commit to either using a "load more" or "infinite scroll" - they conflict so best to just commit to one!

done

[MainlyColors]

Have a button that will initialize infinite scroll/Intersection observer

What is the button for? If it activates infinite scroll doesn't that defeat the purpose of infinite scroll? This would essentially be a load more button

'load more' button is tested to verify that it loads more users when clicked

so is this a load more button?

[DevinCLane]

Have a button that will initialize infinite scroll/Intersection observer

What is the button for? If it activates infinite scroll doesn't that defeat the purpose of infinite scroll? This would essentially be a load more button

'load more' button is tested to verify that it loads more users when clicked

so is this a load more button?

good point, deleting this

[MainlyColors]

With those deleted, I agree with everything on the Tech spec 😁

[devonzara]

Bit of a late thought, but we may also wish to consider how to handle the pagination's state.

For instance, let's assume someone scrolls to page X, clicks on a link and goes to a developer's profile, then clicks their browser's back button. Should they be put back where they left off in the search results or would they have to start over?

Similarly, should users be able to share their search results (including their page/scroll position)?

If users should be able to resume where they left off, then state should be persisted by either:

• storing it in the URL; thus meaning the URL should update as they scroll
• storing it in the client's browser (cookie, session storage, or local storage; depending on how long it should be stored)

If users should be able to share their search results, then having the state in the URL would be useful.

[Caleb-Cohen]

I'd like the back button to keep the same results. Seems annoying to query a search and have 2 profiles you want to look at, you click on one, go back and now can't see the other.

+1 for URL

[DevinCLane]

+1 for storing this in the URL, I'm game.

[timmyichen]

i think for the purposes of this tech spec, we should consider this out of scope for now, and then we can make a separate discussion to talk about how to implement this (it wouldn't need its own tech spec as it's really just a single task with some implementation questions). but just want to avoid blowing up the scope too much as saving state for infinite scroll pagination is no small feat 🙃