Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move internal crawler to crawler project #8

Open
brainless opened this issue Nov 27, 2024 · 0 comments
Open

Move internal crawler to crawler project #8

brainless opened this issue Nov 27, 2024 · 0 comments

Comments

@brainless
Copy link
Member

As a user, I would like to easily access a crawler for the web. This would allow me to easily research content from the web and grow the knowledge graph. Right now, the crawler is in the graph engine's codebase and that should be moved out since it is not a part of the graph.

Engineering

The separate crawler project uses spider-rs but we can continue building with the simple HTTP request based crawler till we need spider-rs. Let us migrate the simple crawler from within the pixlieai_rs to the crawler project.

@brainless brainless added the good first issue Good for newcomers label Nov 27, 2024
@brainless brainless changed the title Separate the internal crawler into the crawler project Move the internal crawler to the crawler project Nov 27, 2024
@brainless brainless changed the title Move the internal crawler to the crawler project Move internal crawler to the crawler project Nov 27, 2024
@brainless brainless changed the title Move internal crawler to the crawler project Move internal crawler to crawler project Nov 27, 2024
@brainless brainless moved this to Backlog in Pixlie AI Nov 28, 2024
@brainless brainless moved this from Backlog to Ready in Pixlie AI Jan 9, 2025
@brainless brainless removed the good first issue Good for newcomers label Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready
Development

No branches or pull requests

1 participant