Skip to content

Commit

Permalink
Merge pull request #498 from conan-io/release/0.33.0
Browse files Browse the repository at this point in the history
Release 0.33.0
  • Loading branch information
czoido authored Jun 10, 2020
2 parents 3d05949 + ea440b2 commit e145b4c
Show file tree
Hide file tree
Showing 12 changed files with 221 additions and 43 deletions.
34 changes: 25 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1106,12 +1106,19 @@ Using **CONAN_CLANG_VERSIONS** env variable in Travis ci or Appveyor:
already stored credentiales in the local cache. Default [False]
- **allow_gcc_minors** Declare this variable if you want to allow gcc >=5 versions with the minor (5.1, 6.3 etc).
- **exclude_vcvars_precommand** For Visual Studio builds, it exclude the vcvars call to set the environment.
- **build_policy**: Default None.
- "never": No build from sources, only download packages.
- "missing": Build only missing packages.
- **build_policy**: Can be None, single value or a list. Default None.
- None: Only Build current package. Equivalent to `--build current_package_ref`
- "never": No build from sources, only download packages. Equivalent to `--build never`
- "missing": Build only missing packages. Equivalent to `--build missing`
- "outdated": Build only missing or if the available package is not built with the current recipe. Useful to upload new configurations, e.j packages for a new compiler without
rebuild all packages.
- "all": Build all requirements.
rebuild all packages. Equivalent to `--build outdated`
- "all": Build all requirements. Equivalent to `--build`
- "cascade": Build from code all the nodes with some dependency being built (for any reason). Equivalent to `--build cascade`
- "some_package" : Equivalent to `--build some_package`
- "pattern\*": will build only the packages with the reference starting with pattern\*. Equivalent to `--build pattern*`
- ["pattern\*", "another_pattern\*"]: will build only the packages with the reference matching these patterns. Equivalent to `--build pattern* --build another_pattern*`
- ["pattern\*", "outdated"]: `--build pattern* --build outdated`
Check [Conan Build policies](https://docs.conan.io/en/latest/mastering/policies.html) for more details.
- **test_folder**: Custom test folder consumed by Conan create, e.j .conan/test_package
- **lockfile**: Custom conan lockfile to be used, e.j. conan.lock. Default [None]
- **conanfile**: Custom conanfile consumed by Conan create. e.j. conanfile.py
Expand Down Expand Up @@ -1150,6 +1157,7 @@ The current commit message can contain special messages:
- **[skip ci]**: Will skip the building of any package (unless `CONAN_IGNORE_SKIP_CI` is set)
- **[build=XXX]**: Being XXX a build policy (see build_policy parameter reference)
- **[build=XXX] [build=YYY]**: Being XXX and YYY the two build policies to use (see build_policy parameter reference)
## Complete ConanMultiPackager methods reference:
Expand Down Expand Up @@ -1256,11 +1264,19 @@ This is especially useful for CI integration.
- **CONAN_EXCLUDE_VCVARS_PRECOMMAND** For Visual Studio builds, it exclude the vcvars call to set the environment.
- **CONAN_BUILD_REQUIRES** You can specify additional build requires for the generated profile with an environment variable following the same profile syntax and separated by ","
i.e ``CONAN_BUILD_REQUIRES: mingw-installer/7.1@conan/stable, pattern: other/1.0@conan/stable``
- **CONAN_BUILD_POLICY**: Default None.
- "never": No build from sources, only download packages.
- "missing": Build only missing packages.
- **CONAN_BUILD_POLICY**: Comma separated list of build policies. Default None.
- None: Only Build current package. Equivalent to `--build current_package_ref`
- "never": No build from sources, only download packages. Equivalent to `--build never`
- "missing": Build only missing packages. Equivalent to `--build missing`
- "outdated": Build only missing or if the available package is not built with the current recipe. Useful to upload new configurations, e.j packages for a new compiler without
rebuild all packages.
rebuild all packages. Equivalent to `--build outdated`
- "all": Build all requirements. Equivalent to `--build`
- "cascade": Build from code all the nodes with some dependency being built (for any reason). Equivalent to `--build cascade`
- "some_package" : Equivalent to `--build some_package`
- "pattern\*": will build only the packages with the reference starting with pattern\*. Equivalent to `--build pattern*`
- "pattern\*,another_pattern\*": will build only the packages with the reference matching these patterns. Equivalent to `--build pattern* --build another_pattern*`
- "pattern\*,outdated": Equivalent to `--build pattern* --build outdated`
Check [Conan Build policies](https://docs.conan.io/en/latest/mastering/policies.html) for more details.
- **CONAN_CONFIG_URL**: Conan config URL be installed before to build e.j https://github.com/bincrafters/conan-config.git
- **CONAN_CONFIG_ARGS**: Conan config arguments used when installing conan config
- **CONAN_BASE_PROFILE**: Apply options, settings, etc. to this profile instead of `default`.
Expand Down
4 changes: 2 additions & 2 deletions cpt/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

__version__ = '0.32.4'
NEWEST_CONAN_SUPPORTED = "1.25.000"
__version__ = '0.33.0'
NEWEST_CONAN_SUPPORTED = "1.26.000"


def get_client_version():
Expand Down
11 changes: 5 additions & 6 deletions cpt/ci_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,19 +66,18 @@ def __init__(self, printer):
self.manager = GenericManager(printer)

def get_commit_build_policy(self):
pattern = "^.*\[build=(\w*)\].*$"
prog = re.compile(pattern)
msg = self.get_commit_msg()
if not msg:
return None
matches = prog.match(msg)
pattern = "\[build=(\w*)\]"
prog = re.compile(pattern)
matches = prog.findall(msg)
if matches:
build_policy = matches.groups()[0]
if build_policy not in ("never", "outdated", "missing", "all"):
raise Exception("Invalid build policy, valid values: never, outdated, missing")
build_policy = matches
return build_policy
return None


def skip_builds(self):
if os.getenv("CONAN_IGNORE_SKIP_CI"):
return False
Expand Down
16 changes: 9 additions & 7 deletions cpt/packager.py
Original file line number Diff line number Diff line change
Expand Up @@ -244,9 +244,10 @@ def __init__(self, username=None, channel=None, runner=None,
self.ci_manager.get_commit_build_policy() or
os.getenv("CONAN_BUILD_POLICY", None))

if build_policy:
if build_policy.lower() not in ("never", "outdated", "missing", "all"):
raise Exception("Invalid build policy, valid values: never, outdated, missing")
# ensure backward compatibility
# build_policy can be list or a string https://github.com/conan-io/conan-package-tools/issues/394
if build_policy and not isinstance(build_policy, list):
build_policy = [x.strip() for x in build_policy.split(',')]

self.build_policy = build_policy

Expand Down Expand Up @@ -508,12 +509,13 @@ def add_common_builds(self, shared_option_name=None, pure_c=True,
continue
elif key not in raw_options_for_building:
del cloned_options[key]
for key in cloned_options.keys():
cloned_options2 = {}
for key, value in cloned_options.items():
# add package reference to the option name
if not key.startswith("{}:".format(reference.name)):
cloned_options["{}:{}".format(reference.name, key)] = cloned_options.pop(key)
cloned_options2["{}:{}".format(reference.name, key)] = value
# combine all options x values (cartesian product)
build_all_options_values = [dict(zip(cloned_options, v)) for v in product(*cloned_options.values())]
build_all_options_values = [dict(zip(cloned_options2, v)) for v in product(*cloned_options2.values())]

builds = self.build_generator.get_builds(pure_c, shared_option_name,
dll_with_static_runtime, reference,
Expand Down Expand Up @@ -807,4 +809,4 @@ def _get_specified_channel(self, channel, reference):
specified_channel = specified_channel.rstrip()
else:
specified_channel = channel or os.getenv("CONAN_CHANNEL", None)
return self._get_channel(specified_channel, self.stable_channel, self.upload_only_when_tag)
return self._get_channel(specified_channel, self.stable_channel, self.upload_only_when_tag)
2 changes: 1 addition & 1 deletion cpt/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
six>=1.10.0, <1.15.0
conan>=1.7.0, <1.26.0
conan>=1.7.0, <1.27.0
tabulate>=0.8.0, <0.9.0
2 changes: 2 additions & 0 deletions cpt/run_in_docker.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ def run():
upload_only_recipe = os.getenv("CPT_UPLOAD_ONLY_RECIPE")
uploader = Uploader(conan_api, remotes_manager, auth_manager, printer, upload_retry)
build_policy = unscape_env(os.getenv("CPT_BUILD_POLICY"))
if build_policy:
build_policy = build_policy.split(",")
test_folder = unscape_env(os.getenv("CPT_TEST_FOLDER"))
reference = ConanFileReference.loads(os.getenv("CONAN_REFERENCE"))

Expand Down
4 changes: 2 additions & 2 deletions cpt/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ def run(self):
name, version, user, channel, _ = self._reference

if self._build_policy:
self._build_policy = [] if self._build_policy == "all" else [self._build_policy]
self._build_policy = [] if self._build_policy == ["all"] else self._build_policy
# https://github.com/conan-io/conan-package-tools/issues/184
with tools.environment_append({"_CONAN_CREATE_COMMAND_": "1"}):
params = {"name": name, "version": version, "user": user,
Expand Down Expand Up @@ -335,7 +335,7 @@ def get_env_vars(self):
ret["CPT_UPLOAD_ENABLED"] = self._upload
ret["CPT_UPLOAD_RETRY"] = self._upload_retry
ret["CPT_UPLOAD_ONLY_RECIPE"] = self._upload_only_recipe
ret["CPT_BUILD_POLICY"] = escape_env(self._build_policy)
ret["CPT_BUILD_POLICY"] = escape_env(self._build_policy.join(",")) if self._build_policy else escape_env(self._build_policy)
ret["CPT_TEST_FOLDER"] = escape_env(self._test_folder)
ret["CPT_CONFIG_URL"] = escape_env(self._config_url)
ret["CPT_CONFIG_ARGS"] = escape_env(self._config_args)
Expand Down
22 changes: 22 additions & 0 deletions cpt/test/integration/basic_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,28 @@ def test_build_policy(self):
conanfile = """from conans import ConanFile
import os
class Pkg(ConanFile):
name = "lib"
version = "1.2"
settings = "os", "compiler", "build_type", "arch"
"""
self.save_conanfile(conanfile)
with tools.environment_append({"CONAN_USERNAME": "lasote"}):
self.packager = ConanMultiPackager(channel="mychannel",
gcc_versions=["6"],
visual_versions=["12"],
archs=["x86", "x86_64"],
build_types=["Release"],
ci_manager=ci_manager)
self.packager.add_common_builds()
self.packager.run()

def test_multiple_build_policy(self):
ci_manager = MockCIManager(build_policy=["lib", "outdated"])
conanfile = """from conans import ConanFile
import os
class Pkg(ConanFile):
name = "lib"
version = "1.2"
Expand Down
121 changes: 121 additions & 0 deletions cpt/test/integration/update_some_deps_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
import unittest

from conans.client.tools import environment_append
from conans.test.utils.tools import TestClient, TestServer
from cpt.test.unit.utils import MockCIManager
from cpt.test.test_client.tools import get_patched_multipackager



class UpdateTest(unittest.TestCase):

conanfile_bar = """from conans import ConanFile
class Pkg(ConanFile):
name = "bar"
version = "0.1.0"
def build(self):
pass
"""

conanfile_foo = """from conans import ConanFile
class Pkg(ConanFile):
name = "foo"
version = "1.0.0"
options = {"shared": [True, False]}
default_options = "shared=True"
def build(self):
pass
"""

conanfile_foo_2 = """from conans import ConanFile
class Pkg(ConanFile):
name = "foo"
version = "1.0.0"
options = {"shared": [True, False]}
default_options = "shared=False"
def build(self):
self.output.info("new foo")
"""

conanfile_foo_3 = """from conans import ConanFile
class Pkg(ConanFile):
name = "qux"
version = "1.0.0"
options = {"shared": [True, False]}
default_options = "shared=False"
def build(self):
self.output.info("qux")
"""

conanfile = """from conans import ConanFile
class Pkg(ConanFile):
name = "foobar"
version = "2.0"
requires = "bar/0.1.0@foo/stable", "foo/1.0.0@bar/testing", "qux/1.0.0@qux/stable"
def build(self):
self.output.warn("BUILDING")
"""

def setUp(self):
self._ci_manager = MockCIManager()
self._server = TestServer(users={"user": "password"},
write_permissions=[("bar/0.1.0@foo/stable", "user"),
("foo/1.0.0@bar/testing", "user"),
("qux/1.0.0@qux/stable", "user")])
self._client = TestClient(servers={"default": self._server},
users={"default": [("user", "password")]})
self._client.save({"conanfile_bar.py": self.conanfile_bar})
self._client.run("export conanfile_bar.py foo/stable")
self._client.save({"conanfile_foo.py": self.conanfile_foo})
self._client.run("export conanfile_foo.py bar/testing")
self._client.save({"conanfile_foo3.py": self.conanfile_foo_3})
self._client.run("export conanfile_foo3.py qux/stable")
self._client.save({"conanfile.py": self.conanfile})

def test_update_some_dependencies(self):
with environment_append({"CONAN_UPLOAD": self._server.fake_url,
"CONAN_LOGIN_USERNAME": "user",
"CONAN_PASSWORD": "password", "CONAN_USERNAME": "user",
"CONAN_UPLOAD_DEPENDENCIES": "all",
"CONAN_UPDATE_DEPENDENCIES": "True"}):

mulitpackager = get_patched_multipackager(self._client, username="user",
channel="testing",
build_policy=["foobar", "bar", "foo", "qux"],
exclude_vcvars_precommand=True,
ci_manager=self._ci_manager)
mulitpackager.add({}, {})
mulitpackager.run()

self.assertIn("Uploading packages for 'foobar/2.0@user/testing'", self._client.out)
self.assertIn("Uploading packages for 'bar/0.1.0@foo/stable'", self._client.out)
self.assertIn("Uploading packages for 'foo/1.0.0@bar/testing'", self._client.out)
self.assertIn("Uploading packages for 'qux/1.0.0@qux/stable'", self._client.out)

# only build and upload foobar
mulitpackager = get_patched_multipackager(self._client, username="user",
channel="testing",
build_policy="foobar",
exclude_vcvars_precommand=True,
ci_manager=self._ci_manager,
conanfile="conanfile.py")
mulitpackager.add({}, {})
mulitpackager.run()
self.assertRegexpMatches(str(self._client.out), r'bar/0.1.0@foo/stable:.* - Cache')
self.assertRegexpMatches(str(self._client.out), r'foo/1.0.0@bar/testing:.* - Cache')
self.assertRegexpMatches(str(self._client.out), r'qux/1.0.0@qux/stable:.* - Cache')

self.assertRegexpMatches(str(self._client.out), r'foobar/2.0@user/testing:.* - Build')

self.assertIn("Uploading packages for 'foobar/2.0@user/testing'", self._client.out)
self.assertNotIn("Uploading packages for 'bar/0.1.0@foo/stable'", self._client.out)
self.assertNotIn("Uploading packages for 'foo/1.0.0@bar/testing'", self._client.out)
self.assertNotIn("Uploading packages for 'qux/1.0.0@qux/stable'", self._client.out)



28 changes: 14 additions & 14 deletions cpt/test/unit/ci_manager_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -231,15 +231,15 @@ def test_build_policy(self):
"TRAVIS_COMMIT_MESSAGE":
"This is a great commit [build=outdated] End."}):
manager = CIManager(self.printer)
self.assertEquals(manager.get_commit_build_policy(), "outdated")
self.assertEquals(manager.get_commit_build_policy(), ["outdated"])
self.assertEquals(manager.get_commit_msg(), "This is a great commit "
"[build=outdated] End.")

with tools.environment_append({"TRAVIS": "1",
"TRAVIS_COMMIT_MESSAGE":
"This is a great commit [build=all] End."}):
manager = CIManager(self.printer)
self.assertEquals(manager.get_commit_build_policy(), "all")
self.assertEquals(manager.get_commit_build_policy(), ["all"])
self.assertEquals(manager.get_commit_msg(), "This is a great commit "
"[build=all] End.")
# Appveyor
Expand All @@ -251,26 +251,26 @@ def test_build_policy(self):
"APPVEYOR_REPO_BRANCH": "mybranch",
}):
manager = CIManager(self.printer)
self.assertEquals(manager.get_commit_build_policy(), "missing")
self.assertEquals(manager.get_commit_build_policy(), ["missing"])

# Raise invalid
# Complex messages
m = "double travis pages again due to timeout, travis taking longer " \
"now [skip appveyor] [build=missing]"
with tools.environment_append({"TRAVIS": "1",
"TRAVIS_COMMIT_MESSAGE": m}):
manager = CIManager(self.printer)
self.assertEquals(manager.get_commit_build_policy(), ["missing"])

# multiple build policies
with tools.environment_append({"APPVEYOR": "1",
"APPVEYOR_PULL_REQUEST_NUMBER": "1",
"APPVEYOR_REPO_COMMIT_MESSAGE": "msg",
"APPVEYOR_REPO_COMMIT_MESSAGE_EXTENDED":
"more [build=joujou] ",
"more [build=missing] [build=pattern] ",
"APPVEYOR_REPO_BRANCH": "mybranch",
}):
manager = CIManager(self.printer)
self.assertRaises(Exception, manager.get_commit_build_policy)

# Complex messages
m = "double travis pages again due to timeout, travis taking longer " \
"now [skip appveyor] [build=missing]"
with tools.environment_append({"TRAVIS": "1",
"TRAVIS_COMMIT_MESSAGE": m}):
manager = CIManager(self.printer)
self.assertEquals(manager.get_commit_build_policy(), "missing")
self.assertEquals(manager.get_commit_build_policy(), ["missing", "pattern"])

def test_bamboo_env_vars(self):
self.assertIsNone(os.getenv('CONAN_LOGIN_USERNAME'))
Expand Down
Loading

0 comments on commit e145b4c

Please sign in to comment.