diff --git a/README.md b/README.md index bd7e7e6..e80aeb7 100644 --- a/README.md +++ b/README.md @@ -1,57 +1,61 @@ -# OnLogs +# OnLogs - Lightweight docker logs web viewer + ![Passing Badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/LbP22/7a0933f8cba0bddbcc95c8b850e32663/raw/onlogs_passing__heads_main.json) ![Coverage Badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/LbP22/7a0933f8cba0bddbcc95c8b850e32663/raw/onlogs_units_coverage__heads_main.json) ![License Badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/LbP22/7a0933f8cba0bddbcc95c8b850e32663/raw/license_MIT.json) +![image](https://github.com/devforth/OnLogs/assets/1838656/38d0f184-3810-4389-a5af-2488b3a51276) + + + ### Benefits -- πŸ”‘ Secure and simple way to access logs without server/ssh accession -- πŸ—οΈ Built with Golang & Svelte.js to extract maximum performance and tiny bundle. Logs storage implemented on top of LevelDB -- 🧸 Setup is easy as docker run command/compose snippet +- πŸ”‘ Simple and secure way to access logs of any Docker containers without server/SSH connection +- πŸ—οΈ Built with Golang & Svelte.js to extract maximum performance and keep the image and RAM footprint as small as possible. Logs storage implemented on top of lightweight LevelDB +- 🧸 Installation is easy as docker runs command/compose snippet. HTTP port exposed and could be routed from Nginx/Traefik/Directly - πŸ–± Get every service realtime logs stream with 1 click -- πŸ“± Check logs from smartphone (insane, but we know it happens that you need it) -- 🧾 Open-Source commercial friendly MIT license +- πŸ“± Check logs from your smartphone (insane, but we know it happens that you need it) +- 🧾 Open-source, commercial-friendly MIT license - πŸ’Ύ Small size of Docker image (~ 13 MB) - πŸ‘₯ Share access to logs with team members, revoke any time ### Features -- πŸ’» One host can be used to view logs from all other hosts in case if you are running Cluster -- πŸ”— Share log messages to collegues via link +- πŸ’» One host can be used to view logs from all other hosts in case you are running Cluster +- πŸ”— Share log messages to colleagues via link - πŸ’½ Clear original docker logs to keep your storage size. - πŸ“Š Error/Info/Debug Statistics -- πŸ”Ž Search through logs (configurable case sensetivity) -- πŸ‘ View parameters (parsing JSON, show local/UTC time for every logline) +- πŸ”Ž Search through logs (configurable case sensitivity) +- πŸ‘ View parameters (parsing JSON, showing local/UTC time for every logline) - πŸ”΄ Realtime logs updating ### Roadmap - πŸ—‚ Grouping hosts -- 🏷 Search & Filter by tags (log status, time) +- 🏷 Search and filter by tags (log status, time) - πŸ”ŒPlugins and internal ability to notify about some event (e.g. notify when Error happens) - πŸ“Š Improved statistics -## Hello world & ussage +## Hello world & usage ### Docker Compose example with traefik ```sh onlogs: image: devforth/onlogs restart: always environment: - - PASSWORD= - - PORT= - # - ONLOGS_PATH_PREFIX=/ if using with path prefix + - ADMIN_USERNAME=admin + - ADMIN_PASSWORD= + - PORT=8798 + # - ONLOGS_PATH_PREFIX=/onlogs if want to use with path prefix - ports: - - : labels: - "traefik.enable=true" - - "traefik.http.routers.onlogs.rule=Host(``)" # if using on subdomain - # - traefik.http.routers.onlogs.rule=PathPrefix(``) # if using with path prefix - - "traefik.http.services.onlogs.loadbalancer.server.port=" + - "traefik.http.routers.onlogs.rule=Host(``)" # if using on subdomain, e.g. https://onlogs.yourdomain.com + # - traefik.http.routers.onlogs.rule=PathPrefix(`/onlogs`) # if want to use with a path prefix, e.g. https://yourdomain.com/onlogs + - "traefik.http.services.onlogs.loadbalancer.server.port=8798" volumes: - /var/run/docker.sock:/var/run/docker.sock - - /var/lib/docker/containers:/var/lib/docker/containers # if you want to delete dublicating logs from docker + - /var/lib/docker/containers:/var/lib/docker/containers # if you want to delete duplicating logs from docker - /etc/hostname:/etc/hostname - onlogs-volume:/leveldb @@ -61,14 +65,14 @@ volumes: ### Docker Run example with traefik ```sh -docker run --restart always -e PASSWORD= -e PORT= \ +docker run --restart always -e ADMIN_USERNAME=admin -e PASSWORD= -e PORT=8798 \ -v /var/run/docker.sock:/var/run/docker.sock:ro \ -v /var/lib/docker/containers:/var/lib/docker/containers \ -v /etc/hostname:/etc/hostname \ -v onlogs-volume:/leveldb \ --label traefik.enable=true \ --label traefik.http.routers.onlogs.rule=Host\(\`\`\) \ - --label traefik.http.services.onlogs.loadbalancer.server.port=2874 devforth/onlogs + --label traefik.http.services.onlogs.loadbalancer.server.port=8798 devforth/onlogs ``` Once done, just go to and login as "admin" with . @@ -76,10 +80,11 @@ Once done, just go to and login as "admin" with . ## Available Environment Options: | Environment Variable | Description | Defaults | Required | |----------------------------|---------------------------------|--------|-----------------| -| PASSWORD | Password for default user | | if `AGENT=false` +| ADMIN_USERNAME | Username for initial user | `admin` | if `AGENT=false` +| ADMIN_PASSWORD | Password for initial user | | if `AGENT=false` | PORT | Port to listen on | `2874` | if `AGENT=false` | JWT_SECRET | Secret for JWT tokens for users | Generates randomly | - | ONLOGS_PATH_PREFIX | Base path if you using OnLogs not on subdomain | | only if using on path prefix -| AGENT | Toggles agent mode. If enabled, there will be no web interface available and all logs will be sent and stored on HOST | `false` | - +| AGENT | Toggles agent mode. If enabled, there will be no web interface available, and all logs will be sent and stored on HOST | `false` | - | HOST | Url to OnLogs host from protocol to domain name. | | if `AGENT=true` -| ONLOGS_TOKEN | Token that will use agent to authorize and connect to HOST | Generates with OnLogs interface | if `AGENT=true` +| ONLOGS_TOKEN | Token that will use an agent to authorize and connect to HOST | Generates with OnLogs interface | if `AGENT=true` diff --git a/application/Dockerfile b/application/Dockerfile index c3603ea..1f47cf2 100644 --- a/application/Dockerfile +++ b/application/Dockerfile @@ -3,7 +3,7 @@ FROM node:16-alpine AS frontbuilder WORKDIR /code/ ADD frontend/package-lock.json . ADD frontend/package.json . -RUN npm install -g npm@latest && npm ci +RUN npm install -g npm:16 && npm ci ADD frontend/. . @@ -12,8 +12,6 @@ RUN npm run build COPY . /code/ FROM alpine -RUN apk add bash curl -# tmp COPY --from=frontbuilder /code/dist/ /backend/dist/ @@ -26,10 +24,6 @@ RUN go mod download \ && go build -o main . FROM alpine -RUN apk add bash curl -# tmp - -EXPOSE 2874 COPY --from=frontbuilder /code/dist/ /dist/ COPY --from=backendbuilder /backend/main /backend/main diff --git a/application/backend/app/agent/agent.go b/application/backend/app/agent/agent.go index 8c2a97b..224e852 100644 --- a/application/backend/app/agent/agent.go +++ b/application/backend/app/agent/agent.go @@ -46,7 +46,7 @@ func SendLogMessage(token string, container string, message_item []string) bool } func TryResend() { - token := os.Getenv("ONLOGS_TOKRN") + token := os.Getenv("ONLOGS_TOKEN") containers, _ := os.ReadDir("leveldb/hosts/" + util.GetHost() + "/containers/") for _, container := range containers { tmpDB := vars.BrokenLogs_DBs[container.Name()] diff --git a/application/backend/app/containerdb/containerdb.go b/application/backend/app/containerdb/containerdb.go index dc0e964..a7d4931 100644 --- a/application/backend/app/containerdb/containerdb.go +++ b/application/backend/app/containerdb/containerdb.go @@ -4,6 +4,7 @@ import ( "fmt" "os" "strings" + "time" "github.com/devforth/OnLogs/app/util" "github.com/devforth/OnLogs/app/vars" @@ -28,6 +29,9 @@ func PutLogMessage(db *leveldb.DB, host string, container string, message_item [ panic("Host is not mentioned!") } location := host + "/" + container + if vars.Statuses_DBs[location] == nil { + vars.Statuses_DBs[location] = util.GetDB(host, container, "statuses") + } if strings.Contains(message_item[1], "ERROR") || strings.Contains(message_item[1], "ERR") || // const statuses_errors = ["ERROR", "ERR", "Error", "Err"]; strings.Contains(message_item[1], "Error") || strings.Contains(message_item[1], "Err") { @@ -49,21 +53,28 @@ func PutLogMessage(db *leveldb.DB, host string, container string, message_item [ } else if strings.Contains(message_item[1], "ONLOGS") { vars.Counters_For_Containers_Last_30_Min[location]["meta"]++ vars.Statuses_DBs[location].Put([]byte(message_item[0]), []byte("meta"), nil) - } else { vars.Counters_For_Containers_Last_30_Min[location]["other"]++ vars.Statuses_DBs[location].Put([]byte(message_item[0]), []byte("other"), nil) } - return db.Put([]byte(message_item[0]), []byte(message_item[1]), nil) + err := db.Put([]byte(message_item[0]), []byte(message_item[1]), nil) + tries := 0 + for err != nil && tries < 10 { + db = util.GetDB(host, container, "logs") + err = db.Put([]byte(message_item[0]), []byte(message_item[1]), nil) + time.Sleep(10 * time.Millisecond) + tries++ + } + if err != nil { + panic(err) + } + return err } func GetLogsByStatus(host string, container string, message string, status string, limit int, startWith string, getPrev bool, include bool, caseSensetivity bool) [][]string { logs_db := util.GetDB(host, container, "logs") db := util.GetDB(host, container, "statuses") - if host != util.GetHost() || vars.ActiveDBs[container] == nil { - defer logs_db.Close() - } iter := db.NewIterator(nil, nil) defer iter.Release() @@ -136,9 +147,6 @@ func GetLogsByStatus(host string, container string, message string, status strin func GetLogs(getPrev bool, include bool, host string, container string, message string, limit int, startWith string, caseSensetivity bool) [][]string { db := util.GetDB(host, container, "logs") - if host != util.GetHost() || vars.ActiveDBs[container] == nil { - defer db.Close() - } iter := db.NewIterator(nil, nil) defer iter.Release() @@ -215,18 +223,15 @@ func DeleteContainer(host string, container string, fullDelete bool) { if vars.ActiveDBs[container] != nil { vars.ActiveDBs[container].Close() - newActiveDB, _ := leveldb.OpenFile("leveldb/hosts/"+host+"/containers/"+container+"/logs", nil) - vars.ActiveDBs[container] = newActiveDB + vars.ActiveDBs[container] = util.GetDB(host, container, "active") } if vars.Statuses_DBs[host+"/"+container] != nil { vars.Statuses_DBs[host+"/"+container].Close() - newStatusesDB, _ := leveldb.OpenFile("leveldb/hosts/"+host+"/containers/"+container+"/statuses", nil) - vars.Statuses_DBs[host+"/"+container] = newStatusesDB + vars.Statuses_DBs[host+"/"+container] = util.GetDB(host, container, "statuses") } if vars.Stat_Containers_DBs[host+"/"+container] != nil { vars.Stat_Containers_DBs[host+"/"+container].Close() - newStatDB, _ := leveldb.OpenFile("leveldb/hosts/"+host+"/containers/"+container+"/statistics", nil) - vars.Statuses_DBs[host+"/"+container] = newStatDB + vars.Statuses_DBs[host+"/"+container] = util.GetDB(host, container, "statistics") } vars.Counters_For_Containers_Last_30_Min[host+"/"+container] = map[string]uint64{"error": 0, "debug": 0, "info": 0, "warn": 0, "meta": 0, "other": 0} } diff --git a/application/backend/app/daemon/daemon.go b/application/backend/app/daemon/daemon.go index 451e2dd..32b4908 100644 --- a/application/backend/app/daemon/daemon.go +++ b/application/backend/app/daemon/daemon.go @@ -41,7 +41,7 @@ func validateMessage(message string) (string, bool) { } func createConnection(containerName string) net.Conn { - connection, _ := net.Dial("unix", "/var/run/docker.sock") + connection, _ := net.Dial("unix", os.Getenv("DOCKER_SOCKET_PATH")) fmt.Fprintf( connection, "GET /containers/"+containerName+"/logs?stdout=true&stderr=true×tamps=true&follow=true&since="+strconv.FormatInt(time.Now().Add(-5*time.Second).Unix(), 10)+" HTTP/1.0\r\n\r\n", @@ -66,7 +66,9 @@ func closeActiveStream(containerName string) { newDaemonStreams = append(newDaemonStreams, stream) } } - vars.ActiveDBs[containerName].Close() + if vars.ActiveDBs[containerName] != nil { + vars.ActiveDBs[containerName].Close() + } vars.ActiveDBs[containerName] = nil vars.Active_Daemon_Streams = newDaemonStreams } @@ -109,11 +111,10 @@ func CreateDaemonToDBStream(containerName string) { reader := bufio.NewReader(connection) readHeader(*reader) - current_db := vars.ActiveDBs[containerName] host := util.GetHost() + current_db := util.GetDB(host, containerName, "logs") createLogMessage(current_db, host, containerName, "ONLOGS: Container listening started!") - lastSleep := time.Now().Unix() defer current_db.Close() for { // reading body logLine, get_string_error := reader.ReadString('\n') @@ -145,16 +146,16 @@ func CreateDaemonToDBStream(containerName string) { c.WriteMessage(1, to_send) } - if time.Now().Unix()-lastSleep > 1 { - time.Sleep(5 * time.Millisecond) - lastSleep = time.Now().Unix() - } + time.Sleep(70 * time.Microsecond) } } // make request to docker socket func makeSocketRequest(path string) []byte { - connection, _ := net.Dial("unix", "/var/run/docker.sock") + connection, err := net.Dial("unix", os.Getenv("DOCKER_SOCKET_PATH")) + if err != nil { + panic(err) + } fmt.Fprintf(connection, "GET /"+path+" HTTP/1.0\r\n\r\n") body, _ := ioutil.ReadAll(connection) @@ -172,14 +173,23 @@ func GetContainersList() []string { json.Unmarshal([]byte(body), &result) var names []string - containersDB, _ := leveldb.OpenFile("leveldb/hosts/"+util.GetHost()+"/containersMeta", nil) - defer containersDB.Close() + + containersMetaDB := vars.ContainersMeta_DBs[util.GetHost()] + if containersMetaDB == nil { + containersMetaDB, err := leveldb.OpenFile("leveldb/hosts/"+util.GetHost()+"/containersMeta", nil) + if err != nil { + panic(err) + } + vars.ContainersMeta_DBs[util.GetHost()] = containersMetaDB + } + containersMetaDB = vars.ContainersMeta_DBs[util.GetHost()] + for i := 0; i < len(result); i++ { name := fmt.Sprintf("%v", result[i]["Names"].([]interface{})[0].(string))[1:] id := result[i]["Id"].(string) names = append(names, name) - containersDB.Put([]byte(name), []byte(id), nil) + containersMetaDB.Put([]byte(name), []byte(id), nil) } return names diff --git a/application/backend/app/db/db.go b/application/backend/app/db/db.go index 93686db..2f50a59 100644 --- a/application/backend/app/db/db.go +++ b/application/backend/app/db/db.go @@ -4,47 +4,46 @@ import ( "time" "github.com/devforth/OnLogs/app/util" + "github.com/devforth/OnLogs/app/vars" "github.com/syndtr/goleveldb/leveldb" ) func CreateOnLogsToken() string { - tokensDB, _ := leveldb.OpenFile("leveldb/tokens", nil) - defer tokensDB.Close() - token := util.GenerateJWTSecret() to_put := time.Now().UTC().Add(24 * time.Hour).String() - tokensDB.Put([]byte(token), []byte(to_put), nil) + err := vars.TokensDB.Put([]byte(token), []byte(to_put), nil) + if err != nil { + vars.TokensDB.Close() + vars.TokensDB, vars.TokensDBErr = leveldb.OpenFile("leveldb/tokens", nil) + + err = vars.TokensDB.Put([]byte(token), []byte(to_put), nil) + if err != nil { + panic(err) + } + } return token } func IsTokenExists(token string) bool { - tokensDB, _ := leveldb.OpenFile("leveldb/tokens", nil) - defer tokensDB.Close() - - iter := tokensDB.NewIterator(nil, nil) + iter := vars.TokensDB.NewIterator(nil, nil) defer iter.Release() iter.First() if string(iter.Key()) == token { - tokensDB.Put([]byte(token), []byte("was used"), nil) + vars.TokensDB.Put([]byte(token), []byte("was used"), nil) return true } for iter.Next() { if string(iter.Key()) == token { - tokensDB.Put([]byte(token), []byte("was used"), nil) + vars.TokensDB.Put([]byte(token), []byte("was used"), nil) return true } } - return false } func DeleteUnusedTokens() { for { - db, r := leveldb.OpenFile("leveldb/tokens", nil) - if r != nil { - panic(r) - } - + db := vars.TokensDB iter := db.NewIterator(nil, nil) for iter.Next() { wasUsed := string(iter.Value()) diff --git a/application/backend/app/routes/routes.go b/application/backend/app/routes/routes.go index 6eb5c08..dc4dad9 100644 --- a/application/backend/app/routes/routes.go +++ b/application/backend/app/routes/routes.go @@ -22,7 +22,6 @@ import ( "github.com/devforth/OnLogs/app/util" "github.com/devforth/OnLogs/app/vars" "github.com/gorilla/websocket" - "github.com/syndtr/goleveldb/leveldb" ) func enableCors(w *http.ResponseWriter) { @@ -40,7 +39,7 @@ func enableCors(w *http.ResponseWriter) { func verifyAdminUser(w *http.ResponseWriter, req *http.Request) bool { username, err := util.GetUserFromJWT(*req) - if username != "admin" { + if username != os.Getenv("ADMIN_USERNAME") { (*w).WriteHeader(http.StatusForbidden) json.NewEncoder(*w).Encode(map[string]string{"error": "Only admin can perform this request"}) return false @@ -128,16 +127,14 @@ func AddLogLine(w http.ResponseWriter, req *http.Request) { if vars.Counters_For_Hosts_Last_30_Min[logItem.Host] == nil { go statistics.RunStatisticForContainer(logItem.Host, logItem.Container) } - location := logItem.Host + "/" + logItem.Container - if vars.Statuses_DBs[location] == nil { - vars.Statuses_DBs[location] = util.GetDB(logItem.Host, logItem.Container, "statuses") + err := containerdb.PutLogMessage(util.GetDB(logItem.Host, logItem.Container, "logs"), logItem.Host, logItem.Container, logItem.LogLine) + if err != nil { + defer w.WriteHeader(http.StatusInternalServerError) + panic(err) } - current_db, _ := leveldb.OpenFile("leveldb/hosts/"+logItem.Host+"/containers/"+logItem.Container+"/logs", nil) - containerdb.PutLogMessage(current_db, logItem.Host, logItem.Container, logItem.LogLine) - defer current_db.Close() to_send, _ := json.Marshal([]string{logItem.LogLine[0], logItem.LogLine[1]}) - for _, c := range vars.Connections[location] { + for _, c := range vars.Connections[logItem.Host+"/"+logItem.Container] { c.WriteMessage(1, to_send) } } @@ -162,6 +159,7 @@ func AddHost(w http.ResponseWriter, req *http.Request) { } vars.AgentsActiveContainers[addReq.Hostname] = addReq.Services + // fmt.Println("New host added: " + addReq.Hostname) need to create separate route for SendUpdate func for _, container := range addReq.Services { os.MkdirAll("leveldb/hosts/"+addReq.Hostname+"/containers/"+container, 0700) } @@ -634,6 +632,12 @@ func EditUser(w http.ResponseWriter, req *http.Request) { decoder := json.NewDecoder(req.Body) decoder.Decode(&loginData) + if loginData.Login == os.Getenv("ADMIN_USERNAME") { + w.Header().Add("Content-Type", "application/json") + json.NewEncoder(w).Encode(map[string]string{"error": "Can't edit admin. Use env variables to change admin username and password"}) + return + } + if !userdb.IsUserExists(loginData.Login) { json.NewEncoder(w).Encode(map[string]string{"error": "No such user"}) return @@ -736,7 +740,7 @@ func DeleteUser(w http.ResponseWriter, req *http.Request) { } decoder := json.NewDecoder(req.Body) decoder.Decode(&loginData) - if loginData.Login == "admin" { + if loginData.Login == os.Getenv("ADMIN_USERNAME") { w.Header().Add("Content-Type", "application/json") json.NewEncoder(w).Encode(map[string]string{"error": "Can't delete admin"}) return diff --git a/application/backend/app/routes/routes_test.go b/application/backend/app/routes/routes_test.go index 0c141c0..c85e2b0 100644 --- a/application/backend/app/routes/routes_test.go +++ b/application/backend/app/routes/routes_test.go @@ -76,6 +76,7 @@ func TestGetHosts(t *testing.T) { Name: "onlogs-cookie", Value: util.CreateJWT("testuser"), }) + os.Setenv("DOCKER_SOCKET_PATH", "/var/run/docker.sock") userdb.CreateUser("testuser", "testuser") rr1 := httptest.NewRecorder() diff --git a/application/backend/app/statistics/statistics.go b/application/backend/app/statistics/statistics.go index 1d237e9..fa9b88b 100644 --- a/application/backend/app/statistics/statistics.go +++ b/application/backend/app/statistics/statistics.go @@ -7,10 +7,11 @@ import ( "github.com/devforth/OnLogs/app/util" "github.com/devforth/OnLogs/app/vars" - "github.com/syndtr/goleveldb/leveldb" ) -func restartStats(host string, container string, current_db *leveldb.DB) { +func restartStats(host string, container string) { + current_db := util.GetDB(host, container, "statistics") + var used_storage map[string]map[string]uint64 var location string if container == "" { @@ -41,15 +42,9 @@ func restartStats(host string, container string, current_db *leveldb.DB) { func RunStatisticForContainer(host string, container string) { location := host + "/" + container vars.Counters_For_Containers_Last_30_Min[location] = map[string]uint64{"error": 0, "debug": 0, "info": 0, "warn": 0, "meta": 0, "other": 0} - if vars.Stat_Containers_DBs[location] == nil { - current_db, _ := leveldb.OpenFile("leveldb/hosts/"+host+"/containers/"+container+"/statistics", nil) - defer current_db.Close() - vars.Stat_Containers_DBs[location] = current_db - } - defer delete(vars.Stat_Containers_DBs, location) - defer restartStats(host, container, vars.Stat_Containers_DBs[location]) + defer restartStats(host, container) for { - restartStats(host, container, vars.Stat_Containers_DBs[location]) + restartStats(host, container) time.Sleep(30 * time.Minute) } } @@ -72,9 +67,6 @@ func GetStatisticsByService(host string, service string, value int) map[string]u searchTo := time.Now().Add(-(time.Hour * time.Duration(value/2))).UTC() var tmp_stats map[string]uint64 current_db := util.GetDB(host, service, "statistics") - if vars.Stat_Containers_DBs[location] == nil { - defer current_db.Close() - } iter := current_db.NewIterator(nil, nil) defer iter.Release() iter.Last() diff --git a/application/backend/app/streamer/streamer.go b/application/backend/app/streamer/streamer.go index db61ee1..f4358f2 100644 --- a/application/backend/app/streamer/streamer.go +++ b/application/backend/app/streamer/streamer.go @@ -11,32 +11,14 @@ import ( "github.com/devforth/OnLogs/app/statistics" "github.com/devforth/OnLogs/app/util" "github.com/devforth/OnLogs/app/vars" - "github.com/syndtr/goleveldb/leveldb" ) func createStreams(containers []string) { for _, container := range vars.DockerContainers { if !util.Contains(container, vars.Active_Daemon_Streams) { go statistics.RunStatisticForContainer(util.GetHost(), container) - newDB, err := leveldb.OpenFile("leveldb/hosts/"+util.GetHost()+"/containers/"+container+"/logs", nil) - if err != nil { - fmt.Println("ERROR: " + container + ": " + err.Error()) - newDB, err = leveldb.RecoverFile("leveldb/hosts/"+util.GetHost()+"/containers/"+container+"/logs", nil) - fmt.Println("INFO: " + container + ": recovering db...") - if err == nil { - fmt.Println("INFO: " + container + ": db recovered!") - } else { - fmt.Println("ERROR: " + container + ": " + err.Error()) - } - } - if vars.Statuses_DBs[util.GetHost()+"/"+container] == nil { - statusesDB, _ := leveldb.OpenFile("leveldb/hosts/"+util.GetHost()+"/containers/"+container+"/statuses", nil) - vars.Statuses_DBs[util.GetHost()+"/"+container] = statusesDB - } - vars.ActiveDBs[container] = newDB vars.Active_Daemon_Streams = append(vars.Active_Daemon_Streams, container) if os.Getenv("AGENT") != "" { - vars.BrokenLogs_DBs[container] = util.GetDB(util.GetHost(), container, "/brokenlogs") go daemon.CreateDaemonToHostStream(container) } else { go daemon.CreateDaemonToDBStream(container) @@ -46,6 +28,11 @@ func createStreams(containers []string) { } func StreamLogs() { + if vars.FavsDBErr != nil || vars.StateDBErr != nil || vars.UsersDBErr != nil { + fmt.Println("ERROR: unable to open leveldb", vars.FavsDBErr, vars.StateDBErr, vars.UsersDBErr) + return + } + vars.DockerContainers = daemon.GetContainersList() if os.Getenv("AGENT") != "" { agent.SendInitRequest(vars.DockerContainers) diff --git a/application/backend/app/userdb/userdb.go b/application/backend/app/userdb/userdb.go index 6a86324..c5b6139 100644 --- a/application/backend/app/userdb/userdb.go +++ b/application/backend/app/userdb/userdb.go @@ -3,6 +3,7 @@ package userdb import ( "encoding/json" "errors" + "os" "strings" "github.com/devforth/OnLogs/app/vars" @@ -27,6 +28,9 @@ func GetUsers() []string { users := []string{} iter := vars.UsersDB.NewIterator(nil, nil) for iter.Next() { + if string(iter.Key()) == os.Getenv("ADMIN_USERNAME") { + continue + } users = append(users, string(iter.Key())) } defer iter.Release() diff --git a/application/backend/app/util/util.go b/application/backend/app/util/util.go index 116964d..1f2bc0a 100644 --- a/application/backend/app/util/util.go +++ b/application/backend/app/util/util.go @@ -46,7 +46,12 @@ func Contains(a string, list []string) bool { } func CreateInitUser() { - vars.UsersDB.Put([]byte("admin"), []byte(os.Getenv("PASSWORD")), nil) + admin_username := os.Getenv("ADMIN_USERNAME") + if admin_username == "" { + admin_username = "admin" + os.Setenv("ADMIN_USERNAME", admin_username) + } + vars.UsersDB.Put([]byte(admin_username), []byte(os.Getenv("ADMIN_PASSWORD")), nil) } func ReplacePrefixVariableForFrontend() { @@ -55,7 +60,6 @@ func ReplacePrefixVariableForFrontend() { fmt.Println("INFO: unable to find 'dist' folder") return } - fmt.Println("INFO: base onlogs prefix is: ", "\""+os.Getenv("ONLOGS_PATH_PREFIX")+"\"") for _, file := range files { if file.IsDir() { dir_files, _ := os.ReadDir("dist/" + file.Name()) @@ -84,16 +88,39 @@ func GetDB(host string, container string, dbType string) *leveldb.DB { res_db = vars.Statuses_DBs[host+"/"+container] } else if dbType == "statistics" { res_db = vars.Stat_Containers_DBs[host+"/"+container] + } else if dbType == "brokenlogs" { + res_db = vars.BrokenLogs_DBs[container] + } + + if res_db != nil { + return res_db } var err error - if res_db == nil { - path := "leveldb/hosts/" + host + "/containers/" + container + "/" + dbType - res_db, err = leveldb.OpenFile(path, nil) - if err != nil { - res_db, _ = leveldb.RecoverFile(path, nil) - } + tries := 0 + path := "leveldb/hosts/" + host + "/containers/" + container + "/" + dbType + res_db, err = leveldb.OpenFile(path, nil) + for (err != nil && res_db == nil) && tries < 10 { + res_db, err = leveldb.RecoverFile(path, nil) + fmt.Println(path, err) + time.Sleep(10 * time.Millisecond) + tries++ + } + + if err != nil { + panic("ERROR: unable to open db for " + host + "/" + container + "/" + dbType + "\n" + err.Error()) + } + + if dbType == "logs" { + vars.ActiveDBs[container] = res_db + } else if dbType == "statuses" { + vars.Statuses_DBs[host+"/"+container] = res_db + } else if dbType == "statistics" { + vars.Stat_Containers_DBs[host+"/"+container] = res_db + } else if dbType == "brokenlogs" { + vars.BrokenLogs_DBs[container] = res_db } + return res_db } @@ -177,9 +204,17 @@ func GetDockerContainerID(host string, container string) string { return "" } - idDB, _ := leveldb.OpenFile("leveldb/hosts/"+host+"/containersMeta", nil) - defer idDB.Close() - iter := idDB.NewIterator(nil, nil) + containersMetaDB := vars.ContainersMeta_DBs[host] + if containersMetaDB == nil { + containersMetaDB, err := leveldb.OpenFile("leveldb/hosts/"+host+"/containersMeta", nil) + if err != nil { + panic(err) + } + vars.ContainersMeta_DBs[host] = containersMetaDB + } + containersMetaDB = vars.ContainersMeta_DBs[host] + + iter := containersMetaDB.NewIterator(nil, nil) defer iter.Release() iter.Last() diff --git a/application/backend/app/vars/vars.go b/application/backend/app/vars/vars.go index a8332e4..47b20dd 100644 --- a/application/backend/app/vars/vars.go +++ b/application/backend/app/vars/vars.go @@ -9,22 +9,30 @@ import ( ) var ( - ActiveDBs = map[string]*leveldb.DB{} - Stat_Containers_DBs = map[string]*leveldb.DB{} - Stat_Hosts_DBs = map[string]*leveldb.DB{} - Statuses_DBs = map[string]*leveldb.DB{} - BrokenLogs_DBs = map[string]*leveldb.DB{} - Active_Daemon_Streams = []string{} - DockerContainers = []string{} - AgentsActiveContainers = map[string][]string{} - ToDelete = map[string][]string{} - Connections = map[string][]websocket.Conn{} + ActiveDBs = map[string]*leveldb.DB{} + Stat_Containers_DBs = map[string]*leveldb.DB{} + Stat_Hosts_DBs = map[string]*leveldb.DB{} + Statuses_DBs = map[string]*leveldb.DB{} + BrokenLogs_DBs = map[string]*leveldb.DB{} + ContainersMeta_DBs = map[string]*leveldb.DB{} + + Active_Daemon_Streams = []string{} + + DockerContainers = []string{} + AgentsActiveContainers = map[string][]string{} + + ToDelete = map[string][]string{} + Connections = map[string][]websocket.Conn{} + Counters_For_Hosts_Last_30_Min = map[string]map[string]uint64{} Counters_For_Containers_Last_30_Min = map[string]map[string]uint64{} - FavsDB, _ = leveldb.OpenFile("leveldb/favourites", nil) - StateDB, _ = leveldb.OpenFile("leveldb/state", nil) - UsersDB, _ = leveldb.OpenFile("leveldb/users", nil) // should i ever close it? - Year = strconv.Itoa(time.Now().UTC().Year()) + + FavsDB, FavsDBErr = leveldb.OpenFile("leveldb/favourites", nil) + StateDB, StateDBErr = leveldb.OpenFile("leveldb/state", nil) + UsersDB, UsersDBErr = leveldb.OpenFile("leveldb/users", nil) + TokensDB, TokensDBErr = leveldb.OpenFile("leveldb/tokens", nil) + + Year = strconv.Itoa(time.Now().UTC().Year()) ) type UserData struct { diff --git a/application/backend/build.sh b/application/backend/build.sh deleted file mode 100644 index 22b37a5..0000000 --- a/application/backend/build.sh +++ /dev/null @@ -1,3 +0,0 @@ -docker build . -t devforth/onlogs -docker push devforth/onlogs - diff --git a/application/backend/main.go b/application/backend/main.go index e65c136..4eb84d7 100644 --- a/application/backend/main.go +++ b/application/backend/main.go @@ -12,8 +12,29 @@ import ( "github.com/joho/godotenv" ) +func init_config() { + if os.Getenv("PORT") == "" { + os.Setenv("PORT", "2874") + } + + if os.Getenv("JWT_SECRET") == "" { + token, err := os.ReadFile("leveldb/JWT_secret") + if err != nil { + os.WriteFile("leveldb/JWT_secret", []byte(os.Getenv("JWT_SECRET")), 0700) + token, _ = os.ReadFile("leveldb/JWT_secret") + } + os.Setenv("JWT_SECRET", string(token)) + } + + if os.Getenv("DOCKER_SOCKET_PATH") == "" { + os.Setenv("DOCKER_SOCKET_PATH", "/var/run/docker.sock") + } + fmt.Println("INFO: OnLogs configs done!") +} + func main() { godotenv.Load(".env") + init_config() if os.Getenv("AGENT") != "" { streamer.StreamLogs() } @@ -24,15 +45,6 @@ func main() { util.ReplacePrefixVariableForFrontend() util.CreateInitUser() - if os.Getenv("JWT_SECRET") == "" { - token, err := os.ReadFile("leveldb/JWT_secret") - if err != nil { - os.WriteFile("leveldb/JWT_secret", []byte(os.Getenv("JWT_SECRET")), 0700) - token, _ = os.ReadFile("leveldb/JWT_secret") - } - os.Setenv("JWT_SECRET", string(token)) - } - pathPrefix := os.Getenv("ONLOGS_PATH_PREFIX") http.HandleFunc(pathPrefix+"/", routes.Frontend) http.HandleFunc(pathPrefix+"/api/v1/addHost", routes.AddHost) @@ -65,5 +77,6 @@ func main() { http.HandleFunc(pathPrefix+"/api/v1/logout", routes.Logout) http.HandleFunc(pathPrefix+"/api/v1/updateUserSettings", routes.UpdateUserSettings) + fmt.Println("Listening on port:", string(os.Getenv("PORT"))+"...") fmt.Println("ONLOGS: ", http.ListenAndServe(":"+string(os.Getenv("PORT")), nil)) } diff --git a/application/build.sh b/application/build.sh new file mode 100644 index 0000000..16d4006 --- /dev/null +++ b/application/build.sh @@ -0,0 +1,2 @@ +docker buildx create --use +docker buildx build --platform=linux/amd64,linux/arm64 --tag "devforth/onlogs:latest" --tag "devforth/onlogs:1.0.4" --push . diff --git a/application/frontend/src/Views/Logs/NewLogsV2.svelte b/application/frontend/src/Views/Logs/NewLogsV2.svelte index 0ce7b54..bf03d8f 100644 --- a/application/frontend/src/Views/Logs/NewLogsV2.svelte +++ b/application/frontend/src/Views/Logs/NewLogsV2.svelte @@ -607,10 +607,10 @@ if (logsContEl) { logsContEl.addEventListener("scroll", function () { - let st = window.pageYOffset || logsContEl.scrollTop; + let st = window.scrollY || logsContEl.scrollTop; if (st > lastScrollTop) { scrollDirection = "down"; - } else { + } else if (st != lastScrollTop) { scrollDirection = "up"; } lastScrollTop = st <= 0 ? 0 : st; // For Mobile or negative scrolling diff --git a/application/frontend/src/Views/Main/Main.svelte b/application/frontend/src/Views/Main/Main.svelte index dd9565e..f31d10e 100644 --- a/application/frontend/src/Views/Main/Main.svelte +++ b/application/frontend/src/Views/Main/Main.svelte @@ -264,7 +264,7 @@
diff --git a/application/frontend/src/lib/ConfirmationMenu/ConfirmationMenu.scss b/application/frontend/src/lib/ConfirmationMenu/ConfirmationMenu.scss index 8e6902d..84c2858 100644 --- a/application/frontend/src/lib/ConfirmationMenu/ConfirmationMenu.scss +++ b/application/frontend/src/lib/ConfirmationMenu/ConfirmationMenu.scss @@ -27,9 +27,6 @@ .boldText { font-weight: 800; } - .confirmationText { - margin: 20px 0 0 0; - } .deleteModalTitle { margin-top: 16px; @@ -41,7 +38,7 @@ display: flex; flex-direction: column; margin-top: 24px; - margin-bottom: 48px; + margin-bottom: 24px; align-items: center; } diff --git a/application/frontend/src/lib/ConfirmationMenu/ConfirmationMenu.svelte b/application/frontend/src/lib/ConfirmationMenu/ConfirmationMenu.svelte index b494fe6..a7cda00 100644 --- a/application/frontend/src/lib/ConfirmationMenu/ConfirmationMenu.svelte +++ b/application/frontend/src/lib/ConfirmationMenu/ConfirmationMenu.svelte @@ -14,7 +14,7 @@ import Checkbox from "../CheckBox/Checkbox.svelte"; import fetchApi from "../../utils/fetch.js"; - let confirmationWord = "I understand that data will be lost"; + let confirmationWord = "I understand that logs will be lost"; let tipsIsVisible = false; let inputValue = ""; let error = false; @@ -79,10 +79,9 @@ /> {#if tipsIsVisible}
- Delete Docker logs - when the option is - disabled - you can only delete duplicates of logs, that onLogs uses to present logs - to you. Logs will be available in docker containers, but not for onLogs. + Delete Docker logs - when the option is set to + "OFF" + logs will be deleted only from onLogs. Logs will be available in docker containers, but not for onLogs. When enabled , each deletion of logs will clear logs from both onLogs and the @@ -101,22 +100,25 @@
-

- {`You want to delete logs. Host: ${ +

+ {`Host: ${ $lastChosenHost ? $lastChosenHost : "host" - } service:${ - $lastChosenService ? $lastChosenService : "service" - } from: `} - - {$confirmationObj.message} + }`}

-

This data will be lost. This action cannot be undone.

+

+ {`Service: ${ + $lastChosenService ? $lastChosenService : "service" + }, from: `} + + {$confirmationObj.message} +

+

This data will be lost. This action cannot be undone.

- Please type:" {confirmationWord}" to confirm.
diff --git a/application/frontend/src/lib/Modal/Modal.scss b/application/frontend/src/lib/Modal/Modal.scss index 234e6a5..f87bbed 100644 --- a/application/frontend/src/lib/Modal/Modal.scss +++ b/application/frontend/src/lib/Modal/Modal.scss @@ -1,5 +1,7 @@ .modalContainer { - min-width: 450px; +// min-width: 40vw; + max-width: 86vw; +// min-width: 40vw; min-height: min-content; background-color: $background-color; diff --git a/application/frontend/src/lib/Stats/Stats.scss b/application/frontend/src/lib/Stats/Stats.scss index 1810f78..394be2f 100644 --- a/application/frontend/src/lib/Stats/Stats.scss +++ b/application/frontend/src/lib/Stats/Stats.scss @@ -2,6 +2,10 @@ padding: 0; margin: 0 0 6px 0; font-size: $main-font-s; + cursor: pointer; + & :hover { + transform: scale(1.02); + } } .statsTittle { diff --git a/application/frontend/src/lib/Stats/Stats.svelte b/application/frontend/src/lib/Stats/Stats.svelte index bb11296..2c85f7f 100644 --- a/application/frontend/src/lib/Stats/Stats.svelte +++ b/application/frontend/src/lib/Stats/Stats.svelte @@ -3,6 +3,7 @@ lastChosenHost, lastChosenService, lastStatsPeriod, + chosenStatus } from "../../Stores/stores.js"; import fetchApi from "../../utils/fetch"; import { navigate } from "svelte-routing"; @@ -90,7 +91,14 @@ return 1; } }) as [key, name]} -
  • +
  • { + if ($chosenStatus !== key) { + chosenStatus.set(key); + } else { + chosenStatus.set(""); + } + }}>

    {key.charAt(0).toUpperCase() + key.slice(1)}

    {name}

    • diff --git a/application/frontend/src/lib/UserMenu/UserMenu.scss b/application/frontend/src/lib/UserMenu/UserMenu.scss index 8e0083f..c4610bd 100644 --- a/application/frontend/src/lib/UserMenu/UserMenu.scss +++ b/application/frontend/src/lib/UserMenu/UserMenu.scss @@ -112,7 +112,7 @@ table { box-sizing: border-box; font-size: $main-font-m; border: 1px solid $lines-color; - margin-top: 42px; + margin-top: 20px; border: 1px solid $text-placeholder-color; color: $text-dark-color; diff --git a/application/frontend/src/lib/UserMenu/UserMenu.svelte b/application/frontend/src/lib/UserMenu/UserMenu.svelte index 22452f4..bd58321 100644 --- a/application/frontend/src/lib/UserMenu/UserMenu.svelte +++ b/application/frontend/src/lib/UserMenu/UserMenu.svelte @@ -161,15 +161,15 @@ class="buttonSpan" on:click={() => { setChosenUserLogin(user); - showUserDeleting(); - }}>