-
-
Notifications
You must be signed in to change notification settings - Fork 482
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
ed8fd79
commit f91fe67
Showing
27 changed files
with
1,259 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
--- | ||
slug: / | ||
--- | ||
|
||
# Introduction | ||
|
||
Hoarder is an open source "Bookmark Everything" app that uses AI for automatically tagging the content you throw at it. The app is built with self-hosting as a first class citizen. | ||
|
||
![Screenshot](https://raw.githubusercontent.com/hoarder-app/hoarder/main/screenshots/homepage.png) | ||
|
||
|
||
## Features | ||
|
||
- 🔗 Bookmark links, take simple notes and store images and pdfs. | ||
- ⬇️ Automatic fetching for link titles, descriptions and images. | ||
- 📋 Sort your bookmarks into lists. | ||
- 🔎 Full text search of all the content stored. | ||
- ✨ AI-based (aka chatgpt) automatic tagging. With supports for local models using ollama! | ||
- 🎆 OCR for extracting text from images. | ||
- 🔖 [Chrome plugin](https://chromewebstore.google.com/detail/hoarder/kgcjekpmcjjogibpjebkhaanilehneje) and [Firefox addon](https://addons.mozilla.org/en-US/firefox/addon/hoarder/) for quick bookmarking. | ||
- 📱 An [iOS app](https://apps.apple.com/us/app/hoarder-app/id6479258022), and an [Android app](https://play.google.com/store/apps/details?id=app.hoarder.hoardermobile&pcampaignid=web_share). | ||
- 📰 Auto hoarding from RSS feeds. | ||
- 🔌 REST API. | ||
- 🌐 Multi-language support. | ||
- 🖍️ Mark and store highlights from your hoarded content. | ||
- 🗄️ Full page archival (using [monolith](https://github.com/Y2Z/monolith)) to protect against link rot. Auto video archiving using [youtube-dl](https://github.com/marado/youtube-dl). | ||
- ☑️ Bulk actions support. | ||
- 🔐 SSO support. | ||
- 🌙 Dark mode support. | ||
- 💾 Self-hosting first. | ||
- [Planned] Downloading the content for offline reading in the mobile app. | ||
|
||
**⚠️ This app is under heavy development and it's far from stable.** | ||
|
||
|
||
## Demo | ||
|
||
You can access the demo at [https://try.hoarder.app](https://try.hoarder.app). Login with the following creds: | ||
|
||
``` | ||
email: [email protected] | ||
password: demodemo | ||
``` | ||
|
||
The demo is seeded with some content, but it's in read-only mode to prevent abuse. |
99 changes: 99 additions & 0 deletions
99
docs/versioned_docs/version-v0.22.0/02-Installation/01-docker.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,99 @@ | ||
# Docker Compose [Recommended] | ||
|
||
### Requirements | ||
|
||
- Docker | ||
- Docker Compose | ||
|
||
### 1. Create a new directory | ||
|
||
Create a new directory to host the compose file and env variables. | ||
|
||
This is where you’ll place the `docker-compose.yml` file from the next step and the environment variables. | ||
|
||
For example you could make a new directory called "hoarder-app" with the following command: | ||
``` | ||
mkdir hoarder-app | ||
``` | ||
|
||
|
||
### 2. Download the compose file | ||
|
||
Download the docker compose file provided [here](https://github.com/hoarder-app/hoarder/blob/main/docker/docker-compose.yml) directly into your new directory. | ||
|
||
``` | ||
wget https://raw.githubusercontent.com/hoarder-app/hoarder/main/docker/docker-compose.yml | ||
``` | ||
|
||
### 3. Populate the environment variables | ||
|
||
To configure the app, create a `.env` file in the directory and add this minimal env file: | ||
|
||
``` | ||
HOARDER_VERSION=release | ||
NEXTAUTH_SECRET=super_random_string | ||
MEILI_MASTER_KEY=another_random_string | ||
NEXTAUTH_URL=http://localhost:3000 | ||
``` | ||
|
||
You **should** change the random strings. You can use `openssl rand -base64 36` in a seperate terminal window to generate the random strings. You should also change the `NEXTAUTH_URL` variable to point to your server address. | ||
|
||
Using `HOARDER_VERSION=release` will pull the latest stable version. You might want to pin the version instead to control the upgrades (e.g. `HOARDER_VERSION=0.10.0`). Check the latest versions [here](https://github.com/hoarder-app/hoarder/pkgs/container/hoarder-web). | ||
|
||
Persistent storage and the wiring between the different services is already taken care of in the docker compose file. | ||
|
||
Keep in mind that every time you change the `.env` file, you'll need to re-run `docker compose up`. | ||
|
||
If you want more config params, check the config documentation [here](/configuration). | ||
|
||
### 4. Setup OpenAI | ||
|
||
To enable automatic tagging, you'll need to configure OpenAI. This is optional though but highly recommended. | ||
|
||
- Follow [OpenAI's help](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key) to get an API key. | ||
- Add the OpenAI API key to the env file: | ||
|
||
``` | ||
OPENAI_API_KEY=<key> | ||
``` | ||
|
||
Learn more about the costs of using openai [here](/openai). | ||
|
||
<details> | ||
<summary>If you want to use Ollama (https://ollama.com/) instead for local inference.</summary> | ||
|
||
**Note:** The quality of the tags you'll get will depend on the quality of the model you choose. | ||
|
||
- Make sure ollama is running. | ||
- Set the `OLLAMA_BASE_URL` env variable to the address of the ollama API. | ||
- Set `INFERENCE_TEXT_MODEL` to the model you want to use for text inference in ollama (for example: `llama3.1`) | ||
- Set `INFERENCE_IMAGE_MODEL` to the model you want to use for image inference in ollama (for example: `llava`) | ||
- Make sure that you `ollama pull`-ed the models that you want to use. | ||
- You might want to tune the `INFERENCE_CONTEXT_LENGTH` as the default is quite small. The larger the value, the better the quality of the tags, but the more expensive the inference will be. | ||
|
||
</details> | ||
|
||
### 5. Start the service | ||
|
||
Start the service by running: | ||
|
||
``` | ||
docker compose up -d | ||
``` | ||
|
||
Then visit `http://localhost:3000` and you should be greeted with the Sign In page. | ||
|
||
### [Optional] 6. Enable optional features | ||
|
||
Check the [configuration docs](/configuration) for extra features to enable such as full page archival, full page screenshots, inference languages, etc. | ||
|
||
### [Optional] 7. Setup quick sharing extensions | ||
|
||
Go to the [quick sharing page](/quick-sharing) to install the mobile apps and the browser extensions. Those will help you hoard things faster! | ||
|
||
## Updating | ||
|
||
Updating hoarder will depend on what you used for the `HOARDER_VERSION` env variable. | ||
|
||
- If you pinned the app to a specific version, bump the version and re-run `docker compose up -d`. This should pull the new version for you. | ||
- If you used `HOARDER_VERSION=release`, you'll need to force docker to pull the latest version by running `docker compose up --pull always -d`. |
19 changes: 19 additions & 0 deletions
19
docs/versioned_docs/version-v0.22.0/02-Installation/02-unraid.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Unraid | ||
|
||
## Docker Compose Manager Plugin (Recommended) | ||
|
||
You can use [Docker Compose Manager](https://forums.unraid.net/topic/114415-plugin-docker-compose-manager/) plugin to deploy Hoarder using the official docker compose file provided [here](https://github.com/hoarder-app/hoarder/blob/main/docker/docker-compose.yml). After creating the stack, you'll need to setup some env variables similar to that from the docker compose installation docs [here](/Installation/docker#3-populate-the-environment-variables). | ||
|
||
## Community Apps | ||
|
||
:::info | ||
The community application template is maintained by the community. | ||
::: | ||
|
||
Hoarder can be installed on Unraid using the community application plugins. Hoarder is a multi-container service, and because unraid doesn't natively support that, you'll have to install the different pieces as separate applications and wire them manually together. | ||
|
||
Here's a high level overview of the services you'll need: | ||
|
||
- **Hoarder** ([Support post](https://forums.unraid.net/topic/165108-support-collectathon-hoarder/)): Hoarder's main web app. | ||
- **Browserless** ([Support post](https://forums.unraid.net/topic/130163-support-template-masterwishxbrowserless/)): The chrome headless service used for fetching the content. Hoarder's official docker compose doesn't use browserless, but it's currently the only headless chrome service available on unraid, so you'll have to use it. | ||
- **MeiliSearch** ([Support post](https://forums.unraid.net/topic/164847-support-collectathon-meilisearch/)): The search engine used by Hoarder. It's optional but highly recommended. If you don't have it set up, search will be disabled. |
48 changes: 48 additions & 0 deletions
48
docs/versioned_docs/version-v0.22.0/02-Installation/03-archlinux.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# Arch Linux | ||
|
||
## Installation | ||
|
||
> [Hoarder on AUR](https://aur.archlinux.org/packages/hoarder) is not maintained by the hoarder official. | ||
1. Install hoarder | ||
|
||
```shell | ||
paru -S hoarder | ||
``` | ||
|
||
2. (**Optional**) Install optional dependencies | ||
|
||
```shell | ||
# meilisearch: for full text search | ||
paru -S meilisearch | ||
# ollama: for automatic tagging | ||
paru -S ollama | ||
# hoarder-cli: hoarder cli tool | ||
paru -S hoarder-cli | ||
``` | ||
|
||
You can use Open-AI instead of `ollama`. If you use `ollama`, you need to download the ollama model. Please refer to: [https://ollama.com/library](https://ollama.com/library). | ||
|
||
3. Set up | ||
|
||
Environment variables can be set in `/etc/hoarder/hoarder.env` according to [configuration page](/configuration). **The environment variables that are not specified in `/etc/hoarder/hoarder.env` need to be added by yourself.** | ||
|
||
4. Enable service | ||
|
||
```shell | ||
sudo systemctl enable --now hoarder.target | ||
``` | ||
|
||
Then visit `http://localhost:3000` and you should be greated with the Sign In page. | ||
|
||
## Services and Ports | ||
|
||
`hoarder.target` include 3 services: `hoarder-web.service`, `hoarder-works.service`, `hoarder-browser.service`. | ||
|
||
- `hoarder-web.service`: Provide hoarder WebUI service, use `3000` port default. | ||
|
||
- `hoarder-works.service`: Provide hoarder works service, no port. | ||
|
||
- `hoarder-browser.service`: Provide browser headless service, use `9222` port default. |
112 changes: 112 additions & 0 deletions
112
docs/versioned_docs/version-v0.22.0/02-Installation/04-kubernetes.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
# Kubernetes | ||
|
||
### Requirements | ||
|
||
- A kubernetes cluster | ||
- kubectl | ||
- kustomize | ||
|
||
### 1. Get the deployment manifests | ||
|
||
You can clone the repository and copy the `/kubernetes` directory into another directory of your choice. | ||
|
||
### 2. Populate the environment variables and secrets | ||
|
||
To configure the app, copy the `.env_sample` to `.env` and change to your specific needs. | ||
|
||
You should also change the `NEXTAUTH_URL` variable to point to your server address. | ||
|
||
Using `HOARDER_VERSION=release` will pull the latest stable version. You might want to pin the version instead to control the upgrades (e.g. `HOARDER_VERSION=0.10.0`). Check the latest versions [here](https://github.com/hoarder-app/hoarder/pkgs/container/hoarder-web). | ||
|
||
To see all available configuration options check the [documentation](https://docs.hoarder.app/configuration). | ||
|
||
To configure the neccessary secrets for the application copy the `.secrets_sample` file to `.secrets` and change the sample secrets to your generated secrets. | ||
|
||
> Note: You **should** change the random strings. You can use `openssl rand -base64 36` to generate the random strings. | ||
### 3. Setup OpenAI | ||
|
||
To enable automatic tagging, you'll need to configure OpenAI. This is optional though but highly recommended. | ||
|
||
- Follow [OpenAI's help](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key) to get an API key. | ||
- Add the OpenAI API key to the `.env` file: | ||
|
||
``` | ||
OPENAI_API_KEY=<key> | ||
``` | ||
|
||
Learn more about the costs of using openai [here](/openai). | ||
|
||
<details> | ||
<summary>[EXPERIMENTAL] If you want to use Ollama (https://ollama.com/) instead for local inference.</summary> | ||
|
||
**Note:** The quality of the tags you'll get will depend on the quality of the model you choose. Running local models is a recent addition and not as battle tested as using openai, so proceed with care (and potentially expect a bunch of inference failures). | ||
|
||
- Make sure ollama is running. | ||
- Set the `OLLAMA_BASE_URL` env variable to the address of the ollama API. | ||
- Set `INFERENCE_TEXT_MODEL` to the model you want to use for text inference in ollama (for example: `mistral`) | ||
- Set `INFERENCE_IMAGE_MODEL` to the model you want to use for image inference in ollama (for example: `llava`) | ||
- Make sure that you `ollama pull`-ed the models that you want to use. | ||
|
||
|
||
</details> | ||
|
||
### 4. Deploy the service | ||
|
||
Deploy the service by running: | ||
|
||
``` | ||
make deploy | ||
``` | ||
|
||
### 5. Access the service | ||
|
||
#### via LoadBalancer IP | ||
|
||
By default, these manifests expose the application as a LoadBalancer Service. You can run `kubectl get services` to identify the IP of the loadbalancer for your service. | ||
|
||
Then visit `http://<loadbalancer-ip>:3000` and you should be greated with the Sign In page. | ||
|
||
> Note: Depending on your setup you might want to expose the service via an Ingress, or have a different means to access it. | ||
#### Via Ingress | ||
|
||
If you want to use an ingress, you can customize the sample ingress in the kubernetes folder and change the host to the DNS name of your choice. | ||
|
||
After that you have to configure the web service to the type ClusterIP so it is only reachable via the ingress. | ||
|
||
If you have already deployed the service you can patch the web service to the type ClusterIP with the following command: | ||
|
||
` kubectl -n hoarder patch service web -p '{"spec":{"type":"ClusterIP"}}' ` | ||
|
||
Afterwards you can apply the ingress and access the service via your chosen URL. | ||
|
||
#### Setting up HTTPS access to the Service | ||
|
||
To access hoarder securely you can configure the ingress to use a preconfigured TLS certificate. This requires that you already have the needed files, namely your .crt and .key file, on hand. | ||
|
||
After you have deployed the hoarder manifests you can deploy your certificate for hoarder in the `hoarder` namespace with this example command. You can name the secret however you want. But be aware that the secret name in the ingress definition has to match the secret name. | ||
|
||
` $ kubectl --namespace hoarder create secret tls hoarder-web-tls --cert=/path/to/crt --key=/path/to/key ` | ||
|
||
If the secret is successfully created you can now configure the Ingress to use TLS via this changes to the spec: | ||
|
||
```` yaml | ||
spec: | ||
tls: | ||
- hosts: | ||
- hoarder.example.com | ||
secretName: hoarder-web-tls | ||
```` | ||
|
||
> Note: Be aware that the hosts have to match between the tls spec and the HTTP spec. | ||
|
||
### [Optional] 6. Setup quick sharing extensions | ||
|
||
Go to the [quick sharing page](/quick-sharing) to install the mobile apps and the browser extensions. Those will help you hoard things faster! | ||
|
||
## Updating | ||
|
||
Edit the `HOARDER_VERSION` variable in the `kustomization.yaml` file and run `make clean deploy`. | ||
|
||
If you have chosen `release` as the image tag you can also destroy the web pod, since the deployment has an ImagePullPolicy set to always the pod always pulls the image from the registry, this way we can ensure that the newest release image is pulled. |
32 changes: 32 additions & 0 deletions
32
docs/versioned_docs/version-v0.22.0/02-Installation/05-pikapods.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
# PikaPods [Paid Hosting] | ||
|
||
:::info | ||
Note: PikaPods shares some of its revenue from hosting hoarder with the maintainer of this project. | ||
::: | ||
|
||
[PikaPods](https://www.pikapods.com/) offers managed paid hosting for many open source apps, including Hoarder. | ||
Server administration, updates, migrations and backups are all taken care of, which makes it well suited | ||
for less technical users. As of Nov 2024, running Hoarder there will cost you ~$3 per month. | ||
|
||
### Requirements | ||
|
||
- A _PikaPods_ account. Can be created for free [here](https://www.pikapods.com/register). You get an initial welcome credit of $5. | ||
|
||
### 1. Choose app | ||
|
||
Choose _Hoarder_ from their [list of apps](https://www.pikapods.com/apps) or use this [direct link](https://www.pikapods.com/pods?run=hoarder). This will either | ||
open a new dialog to add a new _Hoarder_ pod or ask you to log in. | ||
|
||
### 2. Add settings | ||
|
||
There are a few settings to configure in the dialog: | ||
|
||
- **Basics**: Give the pod a name and choose a region that's near you. | ||
- **Env Vars**: Here you can disable signups or set an OpenAI API key. All settings are optional. | ||
- **Resources**: The resources your _Hoarder_ pod can use. The defaults are fine, unless you have a very large collection. | ||
|
||
### 3. Start pod and add user | ||
|
||
After hitting _Add pod_ it will take about a minute for the app to fully start. After this you can visit | ||
the pod's URL and add an initial user under _Sign Up_. After this you may want to disable further sign-ups | ||
by setting the pod's `DISABLE_SIGNUPS` _Env Var_ to `true`. |
Oops, something went wrong.