Skip to content

Commit

Permalink
Remove LXD profile prior to retrying creating LXD instance (#131)
Browse files Browse the repository at this point in the history 
* Remove LXD profile prior to retrying creating LXD instance

* Add missing docstring

* Add unit test

* Fix test

* Fix integration test

* Move integration test to test_charm.py

* Add retry for deletion of LXD profile

* debug

* Debug

* Fix removal of storage profile

* Fix bug

* Fix bug due to github api change

* Remove debug
  • Loading branch information
@yhaliaw
yhaliaw authored Oct 9, 2023
1 parent 5127965 commit 5511464
Show file tree
Hide file tree
Showing 4 changed files with 198 additions and 12 deletions.
95 changes: 84 additions & 11 deletions src-docs/lxd.py.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The LxdClient class offers a low-level interface to isolate the underlying imple
## <kbd>class</kbd> `LxdClient`
LXD client.

<a href="../src/lxd.py#L487"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L532"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `__init__`

Expand Down Expand Up @@ -415,7 +415,7 @@ Create an LXD instance.
## <kbd>class</kbd> `LxdNetworkManager`
LXD network manager.

<a href="../src/lxd.py#L359"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L404"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `__init__`

Expand All @@ -436,7 +436,7 @@ Instantiate the LXD profile manager.

---

<a href="../src/lxd.py#L367"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L412"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `get`

Expand All @@ -458,6 +458,55 @@ Get the LXD network information.
Information on the LXD network.


---

## <kbd>class</kbd> `LxdProfile`
LXD profile.

<a href="../src/lxd.py#L373"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `__init__`

```python
__init__(pylxd_profile: 'Profile')
```

Instantiate the LXD profile.



**Args:**

- <b>`pylxd_profile`</b>: Instance of the pylxd.models.Profile.




---

<a href="../src/lxd.py#L395"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `delete`

```python
delete()
```

Delete the profile.

---

<a href="../src/lxd.py#L390"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `save`

```python
save()
```

Save the current configuration of profile.


---

## <kbd>class</kbd> `LxdProfileManager`
Expand Down Expand Up @@ -540,6 +589,30 @@ Check whether an LXD profile of a given name exists.
**Returns:**
Whether the LXD profile of the given name exists.

---

<a href="../src/lxd.py#L354"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `get`

```python
get(name: 'str') → LxdProfile
```

Get an LXD profile.



**Args:**

- <b>`name`</b>: Name of the LXD profile.



**Raises:**

- <b>`LxdError`</b>: Unable to get the LXD profile with the name.


---

Expand All @@ -548,7 +621,7 @@ An LXD storage pool.

Attrs: name (str): Name of the storage pool. driver (str): Type of driver of the storage pool. used_by (list[str]): LXD instances using the storage pool. config (dict[str, any]): Dictionary of the configuration of the storage pool. managed (bool): Whether LXD manages the storage pool.

<a href="../src/lxd.py#L456"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L501"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `__init__`

Expand All @@ -569,7 +642,7 @@ Instantiate the LXD storage pool.

---

<a href="../src/lxd.py#L478"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L523"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `delete`

Expand All @@ -581,7 +654,7 @@ Delete the storage pool.

---

<a href="../src/lxd.py#L473"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L518"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `save`

Expand All @@ -597,7 +670,7 @@ Save the current configuration of storage pool.
## <kbd>class</kbd> `LxdStoragePoolManager`
LXD storage pool manager.

<a href="../src/lxd.py#L390"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L435"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `__init__`

Expand All @@ -618,7 +691,7 @@ Instantiate the LXD storage pool manager.

---

<a href="../src/lxd.py#L398"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L443"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `all`

Expand All @@ -635,7 +708,7 @@ Get all LXD storage pool.

---

<a href="../src/lxd.py#L432"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L477"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `create`

Expand All @@ -658,7 +731,7 @@ Create an LXD storage pool.

---

<a href="../src/lxd.py#L421"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L466"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `exists`

Expand All @@ -681,7 +754,7 @@ Check if an LXD storage pool exists.

---

<a href="../src/lxd.py#L406"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/lxd.py#L451"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>function</kbd> `get`

Expand Down
45 changes: 45 additions & 0 deletions src/lxd.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,51 @@ def create(
logger.exception("Failed to create LXD profile")
raise LxdError(f"Unable to create LXD profile {name}") from err

def get(self, name: str) -> LxdProfile:
"""Get an LXD profile.
Args:
name: Name of the LXD profile.
Raises:
LxdError: Unable to get the LXD profile with the name.
"""
try:
return self._pylxd_client.profiles.get(name)
except pylxd.exceptions.LXDAPIException as err:
logger.exception("Failed to get LXD profile")
raise LxdError(f"Unable to get LXD profile {name}") from err


class LxdProfile:
"""LXD profile."""

def __init__(
self,
pylxd_profile: pylxd.models.Profile,
):
"""Instantiate the LXD profile.
Args:
pylxd_profile: Instance of the pylxd.models.Profile.
"""
self._pylxd_profile = pylxd_profile

self.name = self._pylxd_profile.name
self.description = self._pylxd_profile.description
self.config = self._pylxd_profile.config
self.devices = self._pylxd_profile.devices
self.used_by = self._pylxd_profile.used_by

def save(self):
"""Save the current configuration of profile."""
self._pylxd_profile.config = self.config
self._pylxd_profile.save()

def delete(self):
"""Delete the profile."""
self._pylxd_profile.delete()


# Disable pylint as public method number check as this class can be extended in the future.
class LxdNetworkManager: # pylint: disable=too-few-public-methods
Expand Down
29 changes: 28 additions & 1 deletion src/runner.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,20 @@ def _create_instance(
"profiles": ["default", "runner", resource_profile],
}

instance = self._clients.lxd.instances.create(config=instance_config, wait=True)
try:
instance = self._clients.lxd.instances.create(config=instance_config, wait=True)
except LxdError:
logger.exception(
"Removing resource profile and storage profile due to LXD instance create failure"
)

# LxdError on creating LXD instance could be caused by improper initialization of
# storage pool. If other runner LXD instance exists then it cannot be the cause.
if not self._clients.lxd.instances.all():
# Removing the storage pool and retry can solve the problem.
self._remove_runner_storage_pool()
raise

self.status.exist = True
return instance

Expand Down Expand Up @@ -299,6 +312,20 @@ def _ensure_runner_storage_pool(self) -> None:
if not self._clients.lxd.storage_pools.exists("runner"):
raise RunnerError("Failed to create runner LXD storage pool")

def _remove_runner_storage_pool(self) -> None:
"""Remove the runner storage pool if exists."""
if self._clients.lxd.storage_pools.exists("runner"):
logger.info("Removing existing runner LXD storage pool.")
runner_storage_pool = self._clients.lxd.storage_pools.get("runner")

# The resource profile needs to be removed first as it uses the storage pool.
for used_by in runner_storage_pool.used_by:
_, profile_name = used_by.rsplit("/", 1)
profile = self._clients.lxd.profiles.get(profile_name)
profile.delete()

runner_storage_pool.delete()

@classmethod
def _get_resource_profile_name(cls, cpu: int, memory: str, disk: str) -> str:
"""Get the LXD profile name for resource limit.
Expand Down
41 changes: 41 additions & 0 deletions tests/integration/test_charm.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from juju.application import Application
from juju.model import Model

from charm import GithubRunnerCharm
from tests.integration.helpers import (
assert_resource_lxd_profile,
get_runner_names,
Expand Down Expand Up @@ -147,3 +148,43 @@ async def test_token_config_changed(model: Model, app: Application, token_alt: s
assert return_code == 0
assert stdout is not None
assert f"GITHUB_TOKEN={token_alt}" in stdout


@pytest.mark.asyncio
@pytest.mark.abort_on_fail
async def test_reconcile_runners_with_lxd_storage_pool_failure(
model: Model, app: Application
) -> None:
"""
arrange: An working application with no runners.
act:
1. a. Set virtual-machines config to 0.
b. Run reconcile_runners action.
c. Delete content in the runner LXD storage directory.
2. a. Set virtual-machines config to 1.
b. Run reconcile_runners action.
assert:
1. No runner should exist.
2. One runner should exist.
"""
unit = app.units[0]

# 1.
await app.set_config({"virtual-machines": "0"})

action = await unit.run_action("reconcile-runners")
await action.wait()
await model.wait_for_idle(status=ACTIVE_STATUS_NAME)
await wait_till_num_of_runners(unit, 0)

exit_code, _ = await run_in_unit(unit, f"rm -rf {GithubRunnerCharm.ram_pool_path}/*")
assert exit_code == 0

# 2.
await app.set_config({"virtual-machines": "1"})

action = await unit.run_action("reconcile-runners")
await action.wait()
await model.wait_for_idle(status=ACTIVE_STATUS_NAME)

await wait_till_num_of_runners(unit, 1)

0 comments on commit 5511464

Please sign in to comment.