Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build(docker): Release 0.5.0 #473

Closed
wants to merge 21 commits into from
Closed
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions kedro-docker/RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@

## Breaking changes to the API

# Release 0.5.0
## Major features and improvements
* Add compatibility with `kedro` 0.19.x

# Release 0.4.0
* Migrate all project metadata to static `pyproject.toml`.
Expand Down
36 changes: 16 additions & 20 deletions kedro-docker/features/docker.feature
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Feature: Docker commands in new projects
Background:
Given I have prepared a config file
And I run a non-interactive kedro new using pandas-iris starter
And I run a non-interactive kedro new using spaceflights-pandas starter
And I have installed the project dependencies
And I have removed old docker image of test project

Expand All @@ -29,26 +29,25 @@ Feature: Docker commands in new projects
Given I have executed the kedro command "docker build"
When I execute the kedro command "docker run"
Then I should get a successful exit code
And I should get a message including "kedro.runner.sequential_runner - INFO - Pipeline execution completed successfully"
And I should get a message including "Pipeline execution completed"

Scenario: Execute docker run with custom base image
When I execute kedro docker build with custom base image
Then I should get a successful exit code
When I execute the kedro command "docker run"
Then I should get a successful exit code
And I should get a message including "kedro.runner.sequential_runner - INFO - Pipeline execution completed successfully"
And I should get a message including "Pipeline execution completed"

Scenario: Execute docker run in parallel mode
Given I have executed the kedro command "docker build"
When I execute the kedro command "docker run --runner=ParallelRunner"
Then I should get a successful exit code
And I should get a message including "kedro.runner.parallel_runner - INFO - Pipeline execution completed successfully"
And I should get a message including "Pipeline execution completed"

Scenario: Use custom UID and GID for Docker image
Given I have executed the kedro command "docker build --uid 10001 --gid 20002"
When I execute the kedro command "docker run"
Then I should get a successful exit code
And I should get a message including "kedro.runner.sequential_runner - INFO - Pipeline execution completed successfully"
When I execute the kedro command "docker cmd id"
Then Standard output should contain a message including "uid=10001(kedro_docker) gid=20002(kedro_group) groups=20002(kedro_group)"

Scenario: Execute docker jupyter notebook target
Given I have executed the kedro command "docker build"
Expand All @@ -75,20 +74,13 @@ Feature: Docker commands in new projects
When I occupy port "8890"
And I execute the kedro command "docker jupyter lab --port 8890"
Then I should get an error exit code
And Standard error should contain a message including "Error: Port 8890 is already in use on the host. Please specify an alternative port number."

Scenario: Execute docker kedro test target
Given I have executed the kedro command "docker build"
When I execute the kedro command "docker cmd kedro test"
Then I should get a successful exit code
And I should get a message including "1 passed"
And I should get a message including "/usr/local/bin/python -m pytest"
And Standard output should contain a message including "Error: Port 8890 is already in use on the host. Please specify an alternative port number."

Scenario: Execute docker cmd without target command
Given I have executed the kedro command "docker build"
When I execute the kedro command "docker cmd"
Then I should get a successful exit code
And I should get a message including "kedro.runner.sequential_runner - INFO - Pipeline execution completed successfully"
And I should get a message including "Pipeline execution completed"

Scenario: Execute docker cmd with non-existent target
Given I have executed the kedro command "docker build"
Expand All @@ -99,13 +91,17 @@ Feature: Docker commands in new projects
Given I have executed the kedro command "docker build"
When I execute the kedro command "docker ipython"
Then I should see messages from docker ipython startup including "An enhanced Interactive Python"
And I should see messages from docker ipython startup including "INFO - Kedro project project-dummy"
And I should see messages from docker ipython startup including "INFO - Defined global variable 'context', 'session', 'catalog' and 'pipelines'"
And I should see messages from docker ipython startup including "Kedro project project-dummy"
And I should see messages from docker ipython startup including "Defined global variable"
And I should see messages from docker ipython startup including "'context'"
And I should see messages from docker ipython startup including "'session'"
And I should see messages from docker ipython startup including "'catalog'"
And I should see messages from docker ipython startup including "'pipelines'"

Scenario: Execute docker run target without building image
When I execute the kedro command "docker run"
Then I should get an error exit code
And Standard error should contain a message including "Error: Unable to find image `project-dummy` locally."
And Standard output should contain a message including "Error: Unable to find image `project-dummy` locally."

Scenario: Execute docker dive target
Given I have executed the kedro command "docker build"
Expand All @@ -123,4 +119,4 @@ Feature: Docker commands in new projects
Scenario: Execute docker dive without building image
When I execute the kedro command "docker dive"
Then I should get an error exit code
And Standard error should contain a message including "Error: Unable to find image `project-dummy` locally."
And Standard output should contain a message including "Error: Unable to find image `project-dummy` locally."
4 changes: 2 additions & 2 deletions kedro-docker/features/docker_with_spark.feature
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Feature: Docker commands in new Spark projects
Background:
Given I have prepared a config file
And I run a non-interactive kedro new using pyspark-iris starter
And I run a non-interactive kedro new using spaceflights-pyspark starter
And I have installed the project dependencies
And I have removed old docker image of test project

Expand All @@ -20,4 +20,4 @@ Feature: Docker commands in new Spark projects
And A new docker image for test project should be created
When I execute the kedro command "docker run"
Then I should get a successful exit code
And I should get a message including "kedro.runner.sequential_runner - INFO - Pipeline execution completed successfully"
And I should get a message including "Pipeline execution completed"
18 changes: 17 additions & 1 deletion kedro-docker/features/steps/cli_steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ def exec_kedro_command(context, command):
@given("I have installed the project dependencies")
def pip_install_dependencies(context):
"""Install project dependencies using pip."""
reqs_path = Path("src", "requirements.txt")
reqs_path = Path("requirements.txt")
res = run(
[context.pip, "install", "-r", str(reqs_path)],
env=context.env,
Expand Down Expand Up @@ -268,6 +268,21 @@ def read_docker_stdout(context, msg):
kill_docker_containers(context.project_name)


@then('Standard output should contain a message including "{msg}"')
def read_docker_stdout_error(context, msg):
"""Read stdout and raise AssertionError if the given message is not there."""

if hasattr(context.result.stdout, "read"):
context.result.stdout = context.result.stdout.read().decode("utf-8")

try:
if msg not in context.result.stdout:
print(context.result.stdout)
assert False, f"Message '{msg}' not found in stdout"
finally:
kill_docker_containers(context.project_name)


@then('Standard error should contain a message including "{msg}"')
def read_docker_stderr(context, msg):
"""Read stderr and raise AssertionError if the given message is not there."""
Expand Down Expand Up @@ -306,6 +321,7 @@ def check_failed_status_code(context):
@then('I should see messages from docker ipython startup including "{msg}"')
def check_docker_ipython_msg(context, msg):
stdout = _get_docker_ipython_output(context)

assert msg in stdout, (
"Expected the following message segment to be printed on stdout: "
f"{msg},\nbut got {stdout}"
Expand Down
2 changes: 1 addition & 1 deletion kedro-docker/kedro_docker/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Kedro plugin for packaging a project with Docker."""

__version__ = "0.4.0"
__version__ = "0.5.0"
2 changes: 1 addition & 1 deletion kedro-docker/kedro_docker/template/Dockerfile.simple
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ ARG BASE_IMAGE=python:3.9-slim
FROM $BASE_IMAGE as runtime-environment

# install project requirements
COPY src/requirements.txt /tmp/requirements.txt
COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache -r /tmp/requirements.txt && rm -f /tmp/requirements.txt

# add kedro user
Expand Down
2 changes: 1 addition & 1 deletion kedro-docker/kedro_docker/template/Dockerfile.spark
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ RUN apt-get update && mkdir -p /usr/share/man/man1 && \
apt-get install -y procps default-jre-headless && rm -rf /var/lib/apt/lists/*

# install project requirements
COPY src/requirements.txt /tmp/requirements.txt
COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache -r /tmp/requirements.txt && rm -f /tmp/requirements.txt

# add kedro user
Expand Down