Skip to content

Commit

Permalink
UPDATE: Secondary readmes (deephaven#4267)
Browse files Browse the repository at this point in the history 
* Bring up-to-speed with Matt's other changes. I'll need to address Chip's comments, which I need to reach out to some other devs about first.

* Updates from Chip's review of Matt's PR

* Updates to proto README

* Updates from Devin and Matt reviews

* Update server/jetty-app/README.md

* Updates from Chip/Matt reviews

---------

Co-authored-by: margaretkennedy <[email protected]>
  • Loading branch information
@jjbrosnan @margaretkennedy
jjbrosnan and margaretkennedy authored Oct 9, 2023
1 parent 77587f4 commit 0c90448
Show file tree
Hide file tree
Showing 3 changed files with 124 additions and 54 deletions.
18 changes: 18 additions & 0 deletions proto/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Deephaven Protobuf

Deephaven's [protobuf](https://protobuf.dev/) files are what drives its client APIs such as [Python](https://deephaven.io/core/client-api/python/), [C++](https://deephaven.io/core/client-api/cpp-examples/getting_started.html), and more using [gRPC](https://grpc.io/). If you wish to build your own client API or change how all client APIs interact with the Deephaven server, this is the place to start.

For Java, proto changes will be automatically reflected when building/running the Java server app. Sections below list files that must be manually generated and committed to source control when proto files change.

## Python/Go/C++ API

```sh
./gradlew :py-client:updateProtobuf :go:updateProtobuf :cpp-client:updateProtobuf
```

## Authorization

See [authorization readme](../authorization/README.md)

## JS API

79 changes: 79 additions & 0 deletions py/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Deephaven Python Source

This folder contains Deephaven Community Core's Python source code, tests, and more. Source code can be found in `server/deephaven` and unit tests can be found in `server/tests`. Other content includes `jpy`, the Python client, wheels, and more.

## Requirements

Development and contribution requires Python 3.8+. We recommend always staying up-to-date on the latest Python version.

## Virtual environments

Deephaven Python development should always be done from a [virtual environment](https://docs.python.org/3/library/venv.html). See [PEP 405](https://peps.python.org/pep-0405/) for the motivation behind virtual environments (venvs). Creating and activating a venv takes two shell commands.

From the root directory of your `deephaven-core` clone:

```sh
python3 -m venv .venv # Create a venv. Only needed once.
source .venv/bin/activate # Activate the virtual environment.
```

A virtual environment only exists in the terminal it's run from. New or other terminal windows or tabs will need to activate the virtual environment again.

Once finished with a virtual environment, exiting is done with a single command.

```sh
deactivate # Exit the venv when finished
```

## Local Development

Using a `pip` editable install is recommended so that Python changes are reflected when your server is restarted. These instructions are for the Python server package in `py/server` but should apply to any other Python packages.

Install the Python server package. This will install the dependencies as well as the autocomplete dependencies.

```sh
DEEPHAVEN_VERSION=0.dev pip install -e "py/server[autocomplete]"
```

After installing, a `pip list` should include `deephaven-core` with an editable project location listed.

Now you can start the Java server by following the instructions [here](../server/jetty-app/README.md). Any time you make Python changes, restart the server to apply them.

## Local Build

You can use these instructions if you do not plan on making frequent changes to your local Python environment or need to try things in a more production-like environment locally.

First, run the following command to build the server and embedded-server if needed. The embedded-server is used to start [pip-installed Deephaven](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/#python-embedded-server).

```sh
./gradlew :py-server:assemble

# If using embedded-server
# You can run both tasks w/ ./gradlew :task1 :task2 if you need both
./gradlew :py-embedded-server:assemble
```

Then run the following commands to install the server package. Note that the `wheel` folder should only contain 1 file, so you should be able to tab-complete the wheel file.

**Note:** Adding the `--no-deps` flag on subsequent installs will skip re-installing other dependencies.

```sh
pip install --force py/server/build/wheel/deephaven_core-<version>-py3-none-any.whl

# If using embedded-server
# You can combine as pip install core.whl server.whl if you need both
pip install --force py/embedded-server/build/wheel/deephaven_server-<version>-py3-none-any.whl
```

**If you make Python server code changes, you will need to re-build and re-install the wheel.**

Start the Java server by following the instructions [here](../server/jetty-app/README.md).

## Tests

Python unit tests must be run from the root directory of the cloned repository.

```sh
cd ..
./gradlew integrations:test-py-deephaven
```
81 changes: 27 additions & 54 deletions server/jetty-app/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,76 +1,40 @@
# server-jetty-app

## Native development
This README is oriented towards getting a server up for local development in a development environment. From a development environment, users can modify source code towards contributing to the project, creating custom capabilities, and more.

This document is oriented towards getting a server up for local development.
If you are interested in running a native production release, please see
[https://deephaven.io/core/docs/how-to-guides/configuration/native-application/](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/).
If you wish to use Deephaven from a production environment, which is simpler but source code cannot be modified, see one of the following documents:

### Groovy Quickstart
- [Quickstart for Docker](https://deephaven.io/core/docs/tutorials/quickstart/)
- [How to configure the Deephaven native application](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/)

```shell
./gradlew server-jetty-app:run -Pgroovy
```

### Python Quickstart

1. Setup virtual environment:

```shell
python -m venv /tmp/my-dh-venv
source /tmp/my-dh-venv/bin/activate
```

1. Build and install wheel

```shell
./gradlew py-server:assemble

# replace with the appropriate <version>
pip install "py/server/build/wheel/deephaven_core-<version>-py3-none-any.whl[autocomplete]"

# To install without the optional `autocomplete` feature, run:
# pip install "py/server/build/wheel/deephaven_core-<version>-py3-none-any.whl"
```

1. Run
This README deals with general development for either the Python or Groovy server-side API. For Python-specific development instructions, see the [Python development README](../../py/README.md).

```shell
./gradlew server-jetty-app:run
```
## Local development

**Note:**
`./gradlew server-jetty-app:run` will incorporate local Java changes on each start. If you are not frequently changing Java code, see the next section.

* This is not an exhaustive guide to managing python environments
* Depending on your OS and how your PATH is setup, you may need to use `python3`, or a path to the explicit python version you want to use
* You may choose to setup a "permanent" virtual environment location
* You'll need to re-install the wheel anytime you are making python code changes that affect the wheel
* `pip` can be a pain if you are trying to (re-)install a wheel with the same version number as before
* A `pip install --force-reinstall --no-deps "py/server/build/wheel/deephaven_core-<version>-py3-none-any.whl[autocomplete]"` may do the trick
* You can install other python packages in your venv using `pip install <some-other-package>`
* You can setup multiple virtual environments, and switch between them as necessary using `source /path/to/other-venv/bin/activate`
* You can de-activate the virtual environment by running `deactivate`
* You can use the `VIRTUAL_ENV` environment variable instead of sourcing / activating virtual environments: `VIRTUAL_ENV=/my/venv ./gradlew server-jetty-app:run`
```shell
./gradlew server-jetty-app:run # Python session (default)
./gradlew server-jetty-app:run -Pdebug # Attach a Java debugger to the Python session on port 5005
./gradlew server-jetty-app:run -Pgroovy # Groovy session
./gradlew server-jetty-app:run -Pgroovy -Pdebug # Attach a Java debugger to the Groovy session on port 5005
```

### Start script
## Development with infrequent changes

To create a more production-like environment, you can create and invoke the start script instead of running via gradle:
To create a more production-like environment, you can create and invoke the start script instead of running via gradle. This is faster if you need to often restart the server without making any changes to Java code (such as Python server development).

```shell
./gradlew server-jetty-app:installDist
./gradlew server-jetty-app:installDist # Run after any Java changes
./server/jetty-app/build/install/server-jetty/bin/start
```


See [https://deephaven.io/core/docs/how-to-guides/configuration/native-application/](https://deephaven.io/core/docs/how-to-guides/configuration/native-application/)
for options when invoking the start script.

### Configuration

The `START_OPTS` environment variable is used to set JVM arguments. For example:

```shell
START_OPTS="-Xmx12g" ./gradlew server-jetty-app:run
START_OPTS="-Xmx12g" ./gradlew server-jetty-app:run # Starts Deephaven with 12gb of heap memory
```

While configuration properties can be inherited via JVM system properties (`-Dmy.property=my.value`), you may prefer to
Expand All @@ -82,12 +46,21 @@ See [config-dir](https://deephaven.io/core/docs/how-to-guides/configuration/nati

See [config-file](https://deephaven.io/core/docs/how-to-guides/configuration/config-file/) for more information on the configuration file format.

### Shutdown

There are multiple ways to shut down the Deephaven server. The easiest is to `ctrl+C` the process. If it's being run in background mode, you can kill it with a `SIGINT`.

```sh
kill -2 <pid>
```

### SSL

By default, the server starts up on all interfaces with plaintext port 10000 (port 443 when SSL is enabled), a token
expiration duration of 5 minutes, a scheduler pool size of 4, and a max inbound message size of 100 MiB.

To bring up a SSL-enabled server on port 8443 with a development key and certificate, you can run:

```shell
./gradlew server-jetty-app:run -Pgroovy -PdevCerts
```
Expand Down Expand Up @@ -155,4 +128,4 @@ outbound.ssl.trust.path=outbound-ca.crt
```

In all of the above cases, the outbound Deephaven-to-Deephaven connections can be configured separately from the
server's inbound configuration.
server's inbound configuration.

0 comments on commit 0c90448

Please sign in to comment.