Issues found and missing documentation for deploying to remote environment

[Thanaphat-Kalaya]

Current Issue

I've encountered challenges while trying to host the WKS Power platform on a remote server. The current documentation mainly focuses on localhost setups, and I struggled to find clear guidance for remote hosting. To improve user experience, a guide or additional resources dedicated to remote hosting would be beneficial.

Proposed Solution

A comprehensive guide for deploying the WKS Power platform on remote servers would be helpful. This guide could include:

• Server requirements and setup configurations.
• Common issues related to remote hosting and recommended solutions.
• Tips for customizing environment variables and .yml/.yaml files.
• Best practices for Keycloak and related security setups.

I'm hoping the guide to outlines best practices for hosting the WKS Power platform on a remote server. Since I've encountered some issues running it on a remote server, I'm hoping that we can work together to find a solution.

What I've Done

To troubleshoot the issues I encountered, I tried the following:

1. Following Installation Guide option 1:
   • Worked only on localhost.
2. Following Installation Guide option 2:
   • Ignored some errors and used the full startup .sh script in the repository to build and run on Docker. The results were similar to option 1.
3. Updating hostnames in .env/.yaml/.yml files and using the full startup script:
   • The Portal didn't redirect to Keycloak's login page due to a realm mismatch. (eg. URL: my vs Keycloak: my.example.com)
4. Editing the Keycloak realm name to match the portal's link:
   • Resolved the login issue but led to CORS issues.
5. Editing the client's origin in Keycloak to "*" (For testing purposes):
   • Resulted in a 401 response from the case engine REST API server.
6. Using Traefik to remove ports from URLs to fix CORS issues:
   • Resulted in a 401 response from the case engine REST API server. (The server logged 404 Not Found: "{"error": "Realm does not exist"}")
   • From the error log, I'm guessing that editing the realm name in Keycloak's setting caused this error. So I dived into Portal's code and found that if the hostname isn't localhost then some sort of substring to the first "." will occur causing realm name mismatch.
7. Editing the portal's code to avoid hostname substring:
   • Resulted in a 403 response from the case engine REST API server with an "insufficient_scope" error.
   • Guessing Traefik path routing may interfere with opa.
8. Stop using Traefik
   • Resulted in a 500 response from the case engine REST API server. (The server logged an Internal Error ... databaseName does not contain '.')
   • Guessing something wrong with DB. Found out that the demo-data-loader is not working as expected and data is not being imported.
   • Investigate through .yml files, found that when using the full hostname as MONGO_DEMO_DATA_LOADER_DATABASE made an error when trying to create the database because of '.' in the hostname. (Previously changed from localhost to my.example.com using auto-replace)
9. Revert the change made that avoids hostname substring & edit MONGO_DEMO_DATA_LOADER_DATABASE as intended
   • The platform is now running. But still have issues like case not being fetch after created and attrachment not being upload etc.

Additional context

While troubleshooting, I faced the following issues:

• Keycloak realm name mismatch, leading to authentication problems. (Cause by my fault .env config)
• CORS errors when connecting to the REST API. (Cause by my fault .env config)
• CORS errors when connecting to the Storage API.
• Inconsistencies with environment variables and docker-compose files.
• Comprehensive Guide for Remote Hosting Configuration

I'm not experienced with Keycloak implementation and Java programming, so I might have misunderstood some aspects. Any guidance or additional resources would be greatly appreciated.

[francav]

I will first move the docs source code into the project repo so that we can collaborate on that @Thanaphat-Kalaya
Then I will add my suggestions on how to address all the items you raised.

WIP: #229

[francav]

First step completed: https://github.com/wkspower/wks-platform/tree/develop/apps/react/docs
The docs are automatically made available https://docs.wkspower.com/ when we integrate changes into the main branch.

[Thanaphat-Kalaya]

Thanks a lot for moving the docs over to the project repo! It’s great to see everything centralized, which definitely helps streamline our collaboration. I’ve taken a quick look at this and everything seems set for us to dive into it together.

While I’m eager to contribute, I must admit I’m still a bit unclear on some aspects of the installation process that we might need to flesh out in the guide. To be honest, I still can't make it fully functional on my remote server. Could you possibly outline where you think my experiences could be most useful, where I messed up, or if there are specific sections you'd like me to focus on? I want to make sure I’m contributing effectively and addressing the right areas.

Looking forward to your insights and working together to make the installation process as smooth as possible for everyone.

[francav]

Definitely @Thanaphat-Kalaya
I am now reviewing the initial description of the issue to provide a suggestion on how to collaborate. Ideally, splitting them on different GH issues so that you can pick the ones you feel comfortable working with.
Hopefully, you shall see them here in a couple of hours.

[francav]

@Thanaphat-Kalaya regarding section "What I've Done/Step 2 - Following Installation Guide option 2". What errors did you need to ignore?

[francav]

@Thanaphat-Kalaya What do you think about breaking down this issue into the following ones? Can you verify if they would address all the issues you raised? I am planning to provide initial info about them and see if we can take it from there.

1. @francav Cretate GH Issue: (documentation) Remote deployment Guide
   1.0. Explain how multi-tenancy is designed and roles/requirements of Keycloak realms names and database names
   1.1 Server requirements and setup configurations.
   1.2 Common issues related to remote hosting and recommended solutions.
   1.3 Tips for customizing environment variables and .yml/.yaml files.
   1.4 Best practices for Keycloak and related security setups.

2. @francav Action Plan -> Cretate GH Issue: (documentation) Improve explanations for Installation Guides(options 1 and 2)

   • Which are the different methods?
   • How do they work?
   • Which method is better to me?
   • which projects are built when using docker?
   • which projects and when you need to manually build?
   • which projects and when you need to manually start?
   • which config files make sense to change on each method?

3. @francav Action Plan -> Create GH Bug: (case-portal) Case list won't reload after case creation unless websocket is active. @Thanaphat-Kalaya Maybe you can assign this one to you? Requires react knowledge only.

4. @francav Action Plan -> Create GH Issue: (documentation) To document websocket publisher and requirements (kafka)

5. Action Plan -> Create GH Bug for - Inconsistencies with environment variables and docker-compose files. @Thanaphat-Kalaya would you create this one?

6. Action Plan -> Create GH Bug with errors found when following Installation Guide option 2. @Thanaphat-Kalaya would you create this one?

7. Action Plan -> Create GH Bug for CORS errors when connecting to the Storage API. @Thanaphat-Kalaya would you create this one?

• Cretate GH Issue: (documentation) Remote deployment Guide (documentation): Installation Option 2: Running with Docker Compose Images #243
• Cretate GH Issue: (documentation) Improve explanations for Installation Guides(options 1 and 2) (documentation) FAQ for Installation Guides (options 1 and 2) #244
• Create GH Bug: (case-portal) Case list won't reload after case creation unless websocket is active (case-portal) Case list won't reload after case creation unless websocket is active #245
• Create GH Issue: (documentation) To document websocket publisher and requirements (kafka) (documentation) To document websocket publisher and requirements (kafka) #246
• Create GH Bug for - Inconsistencies with environment variables and docker-compose files
• Create GH Bug with errors found when following Installation Guide option 2
• Create GH Bug for CORS errors when connecting to the Storage API

[Thanaphat-Kalaya]

@Thanaphat-Kalaya regarding section "What I've Done/Step 2 - Following Installation Guide option 2". What errors did you need to ignore?

Recently, I tried to reproduce the initial errors I mentioned. Interestingly, I found that they no longer occur. It seems like those are related to my fault .env config files causing application startup failures. At that time, I found that the docker-full-startup.sh script could run without throwing errors. So I start debugging my configuration from there.

1. @francav Cretate GH Issue: (documentation) Remote Deployment Guide

The list you provided seems good to me. I'm eager to see the initial draft. Once you've set up the basics, I'd be happy to help refine the details and make sure everything is clear and helpful.

2. @francav Action Plan -> Cretate GH Issue: (documentation) Improve explanations for Installation Guides(options 1 and 2)

I did like to see those as FAQs. This approach would keep the main guide streamlined, preventing users from losing focus on the essential steps, while FAQs can address specific, common issues as they arise. What do you think?

3. @francav Action Plan -> Create GH Bug: (case-portal) Case list won't reload after case creation unless websocket is active. @Thanaphat-Kalaya Maybe you can assign this one to you? Requires react knowledge only.

I must admit, my React & Java knowledge is quite limited for now, and my schedule is pretty packed for the next few months, so I might not be able to tackle this issue right now.

However, I'm happy to assist with other non-coding-related tasks or provide any other support you might need. Once I'm more familiar with the app setup, I can also help you with .env and some docker stuff too. Just let me know how else I can contribute!

5. Action Plan -> Create GH Bug for - Inconsistencies with environment variables and docker-compose files. @Thanaphat-Kalaya would you create this one?

It seems more like a technical debt issue due to the early development process rather than a bug. I'm happy to create the case for this if you think it’s the best approach. Here's the issue #247. Just let me know how you'd like me to proceed further.

6. Action Plan -> Create GH Bug with errors found when following Installation Guide option 2. @Thanaphat-Kalaya would you create this one?

Since it's my fault .env config files causing application startup failures, Once we complete the updated deploying guide, this issue might resolve itself, making a separate bug report may be unnecessary. I suggest we revisit the need for this bug report after we update the guide. Does that sound like a plan to you?

7. Action Plan -> Create GH Bug for CORS errors when connecting to the Storage API. @Thanaphat-Kalaya would you create this one?

It is very likely that they may also be linked to my .env configuration errors. Like with the previous issue, I suggest we hold off on creating a bug report until we've updated the guide.