Merge branch 'main' of https://github.com/enricoros/big-agi

README.md

@@ -13,10 +13,10 @@ Or fork & run on Vercel
 
 [![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fenricoros%2Fbig-agi&env=OPENAI_API_KEY,OPENAI_API_HOST&envDescription=OpenAI%20KEY%20for%20your%20deployment.%20Set%20HOST%20only%20if%20non-default.)
 
-## 🗺️ Explore the Roadmap
+## 👉 [roadmap](https://github.com/users/enricoros/projects/4/views/2)
 
-The development of big-AGI is an open book. Our **[public roadmap](https://github.com/users/enricoros/projects/4/views/2)** is
-live, providing a detailed look at the current and future development of the application.
+big-AGI is an open book; our **[public roadmap](https://github.com/users/enricoros/projects/4/views/2)**
+shows the current developments and future ideas.
 
 - Got a suggestion? [_Add your roadmap ideas_](https://github.com/enricoros/big-agi/issues/new?&template=roadmap-request.md)
 - Want to contribute? [_Pick up a task!_](https://github.com/users/enricoros/projects/4/views/4) - _easy_ to _pro_

docs/changelog.md

@@ -2,10 +2,12 @@
 
 This is a high-level changelog. Calls out some of the high level features batched
 by release.
- - For the live roadmap, please see [the GitHub project](https://github.com/users/enricoros/projects/4/views/2)
+
+- For the live roadmap, please see [the GitHub project](https://github.com/users/enricoros/projects/4/views/2)
 
 ### 1.6.0 - Dec 2023
-- work in progress: [big-AGI open roadmap](https://github.com/users/enricoros/projects/4/views/2), [help here](https://github.com/users/enricoros/projects/4/views/4) 
+
+- work in progress: [big-AGI open roadmap](https://github.com/users/enricoros/projects/4/views/2), [help here](https://github.com/users/enricoros/projects/4/views/4)
 - milestone: [1.6.0](https://github.com/enricoros/big-agi/milestone/6)
 
 ### ✨ What's New in 1.5.0 👊 - Nov 19, 2023
@@ -20,6 +22,17 @@ by release.
 - **Cloudflare OpenAI API Gateway**: Integrate with Cloudflare for a robust API gateway
 - **Helicone for Anthropic**: Utilize Helicone's tools for Anthropic models
 
+For Developers:
+
+- Runtime Server-Side configuration:  https://github.com/enricoros/big-agi/issues/189. Env vars are
+  not required to be set at build time anymore. The frontend will roundtrip to the backend at the
+  first request to get the configuration. See
+  https://github.com/enricoros/big-agi/blob/main/src/modules/backend/backend.router.ts.
+- CloudFlare developers: please change the deployment command to
+  `rm app/api/trpc-node/[trpc]/route.ts && npx @cloudflare/next-on-pages@1`,
+  as we transitioned to the App router in NextJS 14. The documentation in
+  [docs/deploy-cloudflare.md](../docs/deploy-cloudflare.md) is updated
+
 ### 1.4.0: Sept/Oct: scale OUT
 
 - **Expanded Model Support**: Azure and [OpenRouter](https://openrouter.ai/docs#models) models, including gpt-4-32k
@@ -35,7 +48,7 @@ by release.
 - **Backup/Restore** - save chats, and restore them later
 - **[Local model support with Oobabooga server](../docs/config-local-oobabooga)** - run your own LLMs!
 - **Flatten conversations** - conversations summarizer with 4 modes
-- **Fork conversations** - create a new chat, to experiment with different endings
+- **Fork conversations** - create a new chat, to try with different endings
 - New commands: /s to add a System message, and /a for an Assistant message
 - New Chat modes: Write-only - just appends the message, without assistant response
 - Fix STOP generation - in sync with the Vercel team to fix a long-standing NextJS issue

docs/config-browse.md

@@ -0,0 +1,64 @@
+# Browse Functionality in big-AGI 🌐
+
+Allows users to load web pages across various components of `big-AGI`. This feature is supported by Puppeteer-based
+browsing services, which are the most common way to render web pages in a headless environment.
+
+First of all, you need to procure a Puppteer web browsing service endpoint. `big-AGI` supports services like:
+
+- [BrightData](https://brightdata.com/products/scraping-browser) Scraping Browser
+- [Cloudflare](https://developers.cloudflare.com/browser-rendering/) Browser Rendering, or
+- any other Puppeteer-based service that provides a WebSocket endpoint (WSS)
+- **including [your own browser](#your-own-chrome-browser)**
+
+## Configuration
+
+1. **Procure an Endpoint**: Ensure that your browsing service is running and has a WebSocket endpoint available:
+    - this mustbe in the form: `wss://${auth}@{some host}:{port}`
+
+2. **Configure `big-AGI`**:  navigate to **Preferences** > **Tools** > **Browse** and enter the 'wss://...' connection
+   string provided by your browsing service
+
+3. **Enable Features**: Choose which browse-related features you want to enable:
+    - **Attach URLs**: Automatically load and attach a page when pasting a URL into the composer
+    - **/browse Command**: Use the `/browse` command in the chat to load a web page
+    - **ReAct**: Enable the `loadURL()` function in ReAct for advanced interactions
+
+### Server-Side Configuration
+
+You can set the Puppeteer WebSocket endpoint (`PUPPETEER_WSS_ENDPOINT`) in the deployment before running it.
+This is useful for self-hosted instances or when you want to pre-configure the endpoint for all users, and will
+allow your to skip points 2 and 3 above.
+
+Always deploy your own user authentication, authorization and security solution. For this feature, the tRPC
+route that provides browsing service, shall be secured with a user authentication and authorization solution,
+to prevent unauthorized access to the browsing service.
+
+### Your own Chrome browser
+
+***EXPERIMENTAL - UNTESTED*** - You can use your own Chrome browser as a browsing service, by configuring it to expose
+a WebSocket endpoint.
+
+- close all the Chrome instances (on Windows, check the Task Manager if still running)
+- start Chrome with the following command line options (on Windows, you can edit the shortcut properties):
+    - `--remote-debugging-port=9222`
+- go to http://localhost:9222/json/version and copy the `webSocketDebuggerUrl` value
+    - it should be something like: `ws://localhost:9222/...`
+- paste the value into the Endpoint configuration (see point 2 above)
+
+## Usage
+
+Once configured, you can start using the browse functionality:
+
+- **Paste a URL**: Simply paste a URL into the chat, and `big-AGI` will load the page if the Attach URLs feature is enabled
+- **Use /browse**: Type `/browse [URL]` in the chat to command `big-AGI` to load the specified web page
+- **ReAct**: ReAct will automatically use the `loadURL()` function whenever a URL is encountered
+
+## Support
+
+If you encounter any issues or have questions about configuring the browse functionality, join our community on Discord for support and discussions.
+
+[![Official Discord](https://discordapp.com/api/guilds/1098796266906980422/widget.png?style=banner2)](https://discord.gg/MkH4qj2Jp9)
+
+---
+
+Enjoy the enhanced browsing experience within `big-AGI` and explore the web without ever leaving your chat!

docs/environment-variables.md

@@ -43,6 +43,8 @@ PRODIA_API_KEY=
 # Google Custom Search
 GOOGLE_CLOUD_API_KEY=
 GOOGLE_CSE_ID=
+# Browse
+PUPPETEER_WSS_ENDPOINT=
 ```
 
 ## Variables Documentation
@@ -104,6 +106,8 @@ Enable the app to Talk, Draw, and Google things up.
 | `GOOGLE_CSE_ID`          | Google Custom/Programmable Search Engine ID - [Link to PSE](https://programmablesearchengine.google.com/)               |
 | **Text-To-Image**        | [Prodia](https://prodia.com/) is a reliable image generation service                                                    |
 | `PRODIA_API_KEY`         | Prodia API Key - used with '/imagine ...'                                                                               |
+| **Browse**               |                                                                                                                         |
+| `PUPPETEER_WSS_ENDPOINT` | Puppeteer WebSocket endpoint - used for browsing, etc.                                                                  |
 
 ---
 

next.config.mjs

@@ -9,6 +9,11 @@ let nextConfig = {
   //   },
   // },
 
+  // [puppeteer] https://github.com/puppeteer/puppeteer/issues/11052
+  experimental: {
+    serverComponentsExternalPackages: ['puppeteer-core'],
+  },
+
   webpack: (config, _options) => {
     // @mui/joy: anything material gets redirected to Joy
     config.resolve.alias['@mui/material'] = '@mui/joy';