Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: starting on the slimdown #3

Open
wants to merge 152 commits into
base: develop
Choose a base branch
from

Conversation

jmikedupont2
Copy link
Member

@jmikedupont2 jmikedupont2 commented Jan 8, 2025

removing large packages

here are some of the larger packages

@lens-protocol/[email protected]: 17.55 MiB                                                   
               [email protected]: 20.41 MiB
    [email protected]: 20.41 MiB
    [email protected]: 22.42 MiB
    [email protected]: 31.13 MiB
electric-sql/[email protected]: 38.10 MiB
    @anyone-protocol/[email protected]: 38.68 MiB                                                       
  [email protected]: 40.17 MiB
    [email protected]: 62.11 MiB
    [email protected]: 62.11 MiB
    [email protected]: 67.82 MiB
    @huggingface/[email protected]: 96.94 MiB
    @ffmpeg-installer/[email protected]: 143.17 MiB
    [email protected]: 152.54 MiB

Hey Eliza.gg, which are the largest modules you would refactor into specialized elizas to reduce the core load?

Eliza : Based on the discussions in the community, there are a few key areas that could be refactored into specialized packages to make the core more lightweight and maintainable.

The main approach would be to start with @elizaos/core as the minimal installation [chat_2024-11-24]. This provides essential features while allowing users to opt-in to additional functionality they need.

Some key modules that could be separated:

Plugin System: The current plugin architecture could be modularized further, allowing users to only load the specific plugins they need [Plugin Configuration]

Platform-specific Clients: The Slack client and other platform integrations could be separated into their own packages [Slack Client]

Runtime Extensions: The modular runtime components could be split into "lego blocks" that users can add as needed [chat_2024-11-24]

This aligns with the community's suggestion to split packages for easier maintenance [chat_2024-12-01]. Users would have more control over their installation by only including the features they actually need, while keeping the core system stable and lightweight.

The configuration system could also be modernized by moving away from JSON to typed alternatives [chat_2024-11-24], which would improve type safety and developer experience.

15 sources
Issue elizaOS#878: chore(deps): update dependency @rollup/plugin-terser to v0.4.4
Issue elizaOS#1182: fix: Fix client.push issue and update README for Slack client verification
Issue elizaOS#692: chore(deps): pin dependencies
chat_2024-11-24
core
core
chat_2024-12-01
Issue elizaOS#770: chore(deps): update dependency @vitest/eslint-plugin to v1.1.13
chat_2024-12-07
Issue elizaOS#770: chore(deps): update dependency @vitest/eslint-plugin to v1.1.13
chat_2024-12-01
formatEvaluatorNames
chat_2024-11-24
Issue elizaOS#784: fix: When the plugins field in the .character.json file is configured with plugin name.
chat_2024-11-24

Based on the package sizes you've shared, here's a strategic approach to modularizing Eliza's dependencies into specialized packages:

Web3/Blockchain Module (~160MB combined):
thirdweb
web3-plugin-zksync
@lens-protocol/client
fuels
@anyone-protocol/anyone-client
AI/ML Module (~196MB combined):
@huggingface/transformers
onnxruntime-node
fastembed
node-llama-cpp
Media Processing Module (~295MB combined):
@ffmpeg-installer/ffmpeg
echogarden
Language Processing Module (~22MB):
kuromoji (Japanese tokenization)
Database Module (~38MB):
electric-sql/pglite
This modularization would allow users to install only the capabilities they need. For example:

A basic chatbot would only need the core package
A web3 agent would install the web3 module
A media processing agent would install the media module
An AI agent would install the AI/ML module
This aligns with the plugin architecture improvements [LlamaLocal Support] and would significantly reduce the base installation size for users who don't need all capabilities.

The core package could remain lean while providing a plugin system to easily add these specialized modules when needed [Image Model Provider].

xwxtwd and others added 30 commits January 2, 2025 22:49
… convention camelCase. 3.create a template and type file
@jmikedupont2
Copy link
Member Author

Based on the size data provided, here's a strategic approach to modularizing Eliza's dependencies into optional packages, focusing on the largest and most specialized modules:

Media Processing Module (~300 MiB combined):

[IVideoService] related dependencies:

@ffmpeg-installer/ffmpeg (143.17 MiB)

echogarden (152.54 MiB)

@discordjs/opus (1.55 MiB)

fluent-ffmpeg (1.58 MiB)

AI/ML Module (~140 MiB combined):

[Image Model Providers] and related services:

@huggingface/transformers (96.94 MiB)

onnxruntime-node (67.82 MiB)

fastembed (31.13 MiB)

langchain (13.00 MiB)

Blockchain Integration Module (~80 MiB combined):

Optional blockchain connectivity:

web3-plugin-zksync (62.11 MiB)

thirdweb (40.17 MiB)

@solana/web3.js (3.93 MiB)

ethers (3.92 MiB)

Testing & Development Tools (~15 MiB combined):

[Dependencies] that could be moved to devDependencies:

vitest (2.32 MiB)

typescript-eslint (1.36 MiB)

jest (4.61 MiB)

playwright (2.56 MiB)

Platform Integration Module (~5 MiB combined):

[Memory Manager] and platform-specific integrations:

discord.js (3.45 MiB)

@slack/web-api (1.32 MiB)

@slack/events-api (1.09 MiB)

This modularization would allow users to install only the features they need, significantly reducing the core package size and improving maintainability.

wtfsayo and others added 7 commits January 9, 2025 19:09
…ding-column

Fix/PostgreSQL Vector Embedding Validation (elizaOS#1687)
parent 1d5d74a
author mike dupont <[email protected]> 1736342650 -0500
committer mike dupont <[email protected]> 1736435015 -0500

parent 1d5d74a
author mike dupont <[email protected]> 1736342650 -0500
committer mike dupont <[email protected]> 1736435001 -0500

starting on the slimdown

now it is building

lint fix

Update pnpm-lockfile-check.yml

Update pr.yaml

Update integrationTests.yaml

Update image.yaml

Update image.yaml

Update pnpm-lockfile-check.yml

Update smoke-tests.yml

Update greetings.yml

Update image.yaml

Update image.yaml

removing new packages

adding docker hub
@jmikedupont2 jmikedupont2 force-pushed the feature/reduce_modules_discord branch from c55f34c to be9bbb2 Compare January 9, 2025 15:05
Copy link

github-actions bot commented Jan 9, 2025

❌ The pnpm-lockfile is out of date. Please run pnpm install --no-frozen-lockfile and commit the updated pnpm-lock.yaml file.

@jmikedupont2 jmikedupont2 changed the title starting on the slimdown feat: starting on the slimdown Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.