Skip to content

Cloudflare scraper and cralwer written in Async, In-place library for HTTPX. Crawl website that has cloudflare enabled, easier than ever!

License

Notifications You must be signed in to change notification settings

ManiMozaffar/cfcrawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

96f95ae Β· Dec 2, 2023

History

3 Commits
Dec 2, 2023
Dec 2, 2023
Nov 29, 2023
Nov 29, 2023
Nov 29, 2023
Nov 29, 2023
Nov 29, 2023
Nov 29, 2023
Nov 29, 2023
Nov 29, 2023
Nov 29, 2023
Nov 29, 2023
Dec 2, 2023
Nov 29, 2023
Dec 2, 2023
Dec 2, 2023

Repository files navigation

cfcrawler

Release Build status codecov Commit activity License

Cloudflare scraper and cralwer written in Async, In-place library for HTTPX. Crawl website that has cloudflare enabled, easier than ever!

Getting started

To use library, simply replace your aiohttp client with ours!

from cfcrawler import AsyncClient

async def get(url):
    client = AsyncClient()
    await client.get(url)

You can also rotate user agents

from cfcrawler import AsyncClient

client = AsyncClient()
client.rotate_useragent()

You can also specify which browser you want to use

from cfcrawler.types import Browser
from cfcrawler import AsyncClient

AsyncClient(browser=Browser.CHROME)

You can also use asyncer to syncify the implementation

from cfcrawler import AsyncClient
from asyncer import syncify

def get(url):
    client = AsyncClient()
    syncify(client.get)(url)

Coming Next

  1. CF JS Challenge solver
  2. Captcha solver integration (2Captcha and etc)

Contribution

I'll work on this library in few months, I don't have free time right now, but feel free to contribute. I'll check and test the PRs myself!

About

Cloudflare scraper and cralwer written in Async, In-place library for HTTPX. Crawl website that has cloudflare enabled, easier than ever!

Resources

License

Stars

Watchers

Forks

Packages

No packages published