v1.5.0 release

Full Changelog

New Features

• Add clientset for MPIJob, PytorchJob, MXJob, and XGBoostJob #1610 (tenzen-y)
• Add all generation tools to Makefile #1609 (johnugeorge)
• Adding MPI python sdk #1608 (johnugeorge)
• Adding XGboost Python sdk #1607 (johnugeorge)
• Generating MPI python sdk #1606 (johnugeorge)
• Update k8s dependencies to v0.24.1 #1604 (johnugeorge)
• Migrate test framework to GHA #1603 (johnugeorge)
• Add mpi in update-codegen.sh #1600 (ggaaooppeenngg)
• MXNet SDK with Status check fix #1618 (johnugeorge)

Bug Fixes

• fix: MPIJob worker still running when NotEnoughResources #1621 (hackerboy01)
• fix comments for pytorch-controller #1620 (hackerboy01)
• fix: requeue when expire time is not up yet #1614 (Garrybest)
• Look for fully-qualified job role label in Python sdk #1588 (person142)
• fix torch env typo #1573 (kuizhiqing)
• Restart job on failure for Always,OnFailure Policy #1572 (georgkaleido)
• Increase success threshold #1568 (haoxins)
• update status.startTime for pytorchjob and xgboostjob #1567 (cheimu)
• fix: add mpijobs to kubeflow training role #1565 (henrysecond1)
• fix Pytorjob status inaccuracy when task replica scale down #1593 (PeterChg)
• fix: MPIJob cannot use gang-scheduling when --enable-gang-scheduling is set #1557 (cheimu)
• fix api reader issue #1551 (zw0610)
• fix label and CleanPodPolicy for mpi-controller #1550 (zw0610)
• fix UpdateJobStatusInApiServer when gang-scheduling is enabled #1549 (zw0610)
• fix: add namespace filtering when getting pods/services for jobs #1545 (henrysecond1)
• fix: set mpijob runPolicy.cleanPodPolicy to default none #1554 (cheimu)

Misc

• Update training controller image to latest #1625 (johnugeorge)
• Update SDK version to 1.5.0 #1624 (johnugeorge)
• Upgrade common to v0.4.3 #1623 (johnugeorge)
• Adding GHA for automatic image build and push #1615 (johnugeorge)
• Remove presubmit test depending on optional-test-infra #1596 (aws-kf-ci-bot)
• chore: stop action on first fail #1595 (jasonliu747)
• update img url in design doc #1591 (zw0610)
• Remove uncalled mpi-controller DeletePodsAndServices() #1558 (cheimu)
• Update MPIJob unit tests to use spec.runPolicy.cleanPodPolicy #1556 (cheimu)
• Remove table-logger dependency #1544 (person142)
• Bump pyyaml from 5.1 to 5.4 in /py/kubeflow/tf_operator #1542 (dependabot[bot])

v1.5.0-rc.0 release

Full Changelog

Closed issues:

• MPIJob worker still running when NotEnoughResources with enable-gang-scheduling==true? #1617
• unable to fetch TFJob when I use client.go run tfjob #1612
• Pytorchjob dist-mnist no training logs #1601
• kubectl get tfjob -o yaml, but not status output #1598
• missing image in tf_job_design_doc.md #1590
• Labels in Python client are out of date #1587
• PyTorchJob Pods "Not Ready" After Completing Training #1577
• cannot use "github.com/go-openapi/spec".Schema{...} (type "github.com/go-openapi/spec".Schema) as type "k8s.io/kube-openapi/pkg/validation/spec".Schema in field value #1576
• PyTorchJob: OnFailure Policy won't handle pod failure gracefully #1570
• pytorchjob doesn't have status.startTIme. #1566
• Optional-test-infra Deprecation Notice - Training #1561
• Should we update MPIJob unit test CleanPodPolicy field? #1555
• --enable-gang-scheduling=true doesn't work for MPIJob #1548
• PyTorchJob fails when creating a task with a different namespace but the same name #1543
• Reconcile PyTorchJob error: PyTorchJob.status.replicaStatuses: Invalid value: "null" after enable-gang-scheduling #1538
• Job TTLs not working #1533
• Support PodGroup in scheduler-plugins/coscheduling #1518
• support elastic training #1515
• Modified the configuration of RootLogger #1514
• Add checking import order in CI #1510
• Scale down of pytorchJob cause workers pod to restart #1509
• Support label selector based success/failure conditions #1507
• [feat] Support SuccessPolicy in PyTorchJob #1505
• pytorch elastic scheduler error #1504
• Could you add the example of MPIJob in this repository #1502
• [Feature] Create a Informer/ClientSet for PyTorch Jobs #1499
• [feature] Make init container injection logic availabel to all jobs #1498
• Roadmaps for 1.4 release #1496
• [bug] (MpiJob)Init container KubectlDeliveryImage should remain the ability that it can be specified from container parameters or environment variables. #1494
• Reconcile PyTorch Job error Operation cannot be fulfilled on pytrochjobs.kubeflow.org #1492
• Python PytorchJob: no attribute openapi_types for example code #1481
• PyTorch DistributedDataParallel training with multi nodes #1475
• Installing kubeflow-training breaks import for other kubeflow packages (katib, fairing, etc.) #1471
• Deprecate ksonnet and use python/golang to submit jobs #1468
• Help Wanted in ParameterServerStrategy Example. #1459
• Bug: SomeTimes Coredumped using tfjob #1456
• [question] PyTorchJob MNIST example training speed #1454
• tfjob status not match when EnableDynamicWorker set true #1452
• training-operator set scheduler error #1447
• [sdk]: Replace TableLogger component in the SDK for better support with ipykernel>=6.x #1446
• SDK: wait_for_job reports typeError #1445
• Update prometheus monitoring doc #1443
• Master branch should provide a nightly image #1433
• Clean up test folder before testing #1429
• Clean up TF specific docs #1424
• [feature] Support SchedulingPolicy in PyTorchJob #1414
• Hyperlinks in the "Overview" section is incorrect/not found #1411
• add workqueue metric #1407
• Validation fails for MXJob Tune example #1402
• Rate exceeded for aws ecr image #1400
• change layout to follow the standard of kubebuilder? #1397
• [example] kubeflow/tf-dist-mnist-test:1.0 is missing in v1.2-branch examples/v1/dist-mnist #1393
• Update kubeflow/website for 1.4 release #1392
• Cut beta release of tf-operator for 1.4 release #1385
• "invalid memory address or nil pointer dereference" #1382
• some questions about job sync #1379
• Provides a default Grafana dashboard #1376
• [feature] Support different PS/worker types #1369
• Need to copy all (mainly pytorch) framework's example dir to tf-operator/examples #1366
• Add more CRD validations markers to block invalid job on client apply #1363
• Update presubmit and post submit job triggers #1354
• Optimize post submit jobs flow #1353
• Enable leader election in controller manager using controllermanagerconfig #1350
• Support mpi jobs in universal operator #1345
• post-submit job failure in master branch #1343
• Improve observability of universal operator #1340
• Best practice to organize main.go and Dockerfile? #1333
• Should training operator keep clientset in the same repository? #1332
• Test image has incorrect tag? #1329
• Prepare e2e tests for all frameworks #1323
• Reduce e2e replica-restart-policy-tests running time #1319
• Improve logs structure by consolidating libs from controller runtime and controllers #1313
• Enable tests for all frameworks #1311
• [bug] The pod wil be recreated until the expectation expires #1306
• Upgrade CRDs to apiextensions.k8s.io/v1 #1304
• Add role details as new columns to kubectl get jobs output for CRD. #1301
• How to handle long pending pods in a TF-job? #1282
• Could you release a new version of Python SDK #1279
• Update swagger.json schema for TFJobSpec to include RunPolicy [#1278](https://github.com/kubeflow...

v1.4.0

Full Changelog

Merged pull requests:

• extends path in __init__.py for SDK correctly #1531 (cakeislife100)
• Update manifests with latest image tag #1527 (johnugeorge)
• add option for mpi kubectl delivery #1525 (zw0610)
• restore option namespace in launch arguments #1524 (zw0610)
• remove unused scripts #1521 (zw0610)
• remove ChanYiLin from approvers #1513 (ChanYiLin)
• add StacktraceLevel for zapr #1512 (qiankunli)
• add unit tests for tensorflow controller #1511 (zw0610)
• add the example of MPIJob #1508 (hackerboy01)
• Added 2022 roadmap and migrated previous roadmap from kubeflow/common #1500 (terrytangyuan)
• Fix a typo in mpi controller log #1495 (LuBingtan)
• feat(pytorch): Add init container config to avoid DNS lookup failure #1493 (gaocegege)
• chore: Fix GitHub Actions script #1491 (tenzen-y)
• chore: Fix missspell in tfjob #1490 (tenzen-y)
• chore: Update OWNERS #1489 (gaocegege)
• Bump jinja2 from 2.10.1 to 2.11.3 in /py/kubeflow/tf_operator #1487 (dependabot[bot])
• fix comments for mpi-controller #1485 (hackerboy01)
• add expectation-related functions for other resources used in mpi-controller #1484 (zw0610)
• Add MPI job to README now that it's supported #1480 (terrytangyuan)
• add mpi doc #1477 (zw0610)
• Set Go version of base image to 1.17 #1476 (tenzen-y)
• update label for tf-controller #1474 (zw0610)
• Add Akuity to the list of adopters #1473 (terrytangyuan)
• Add PR template with doc checklist #1470 (andreyvelich)
• Add e2e failure debugging guidance #1469 (Jeffwan)
• chore: Add .gitattributes to ignore Jsonnet test code for linguist #1463 (terrytangyuan)
• Migrate additional examples from xgboost-operator #1461 (terrytangyuan)
• Minor edits to README.md #1460 (terrytangyuan)
• add mpi-operator(v1) to the unified operator #1457 (hackerboy01)
• fix tfjob status when enableDynamicWorker set true #1455 (zw0610)
• feat(pytorch): Support elastic training #1453 (gaocegege)
• fix: generate printer columns for job crds #1451 (henrysecond1)
• Fix README typo #1450 (davidxia)
• consistent naming for better readability #1449 (pramodrj07)
• Fix set scheduler error #1448 (qiankunli)
• Add CI to run the tests for Go #1440 (tenzen-y)
• fix: Add missing retrying package that failed the import #1439 (terrytangyuan)
• Generate a single swagger.json file for all frameworks #1437 (alembiewski)
• Update links and files with the new URL #1434 (andreyvelich)
• chore: update CHANGELOG.md #1432 (Jeffwan)
• Add acknowledgement section in README to credit all contributors #1422 (terrytangyuan)
• Add Cisco to Adopters List #1421 (andreyvelich)
• Add Python SDK for Kubeflow Training Operator #1420 (alembiewski)
• docs: Move myself to approvers #1419 (terrytangyuan)
• fix hyperlinks in the 'overview' section #1418 (pramodrj07)
• docs: Migrate adopters of all operators to this repo #1417 (terrytangyuan)
• Feature/support pytorchjob set queue of volcano #1415 (qiankunli)
• Bump controller-tools to 0.6.0 and enable GenerateEmbeddedObjectMeta #1409 (Jeffwan)
• Update scripts to generate sdk for all frameworks #1389 (Jeffwan)

Closed issues:

• Question: What is the recommended way for Data Scientists to run a distributed training job #1535
• Restore KUBEFLOW_NAMESPACE options #1522
• Improve test coverage #1497
• swagger.json missing Pytorchjob.Spec.ElasticPolicy #1483
• [bug] Missing init container in PyTorchJob #1482
• PytorchJob DDP training will stop if I delete a worker pod #1478
• Write down e2e failure debug process #1467
• How can i add the Priorityclass to the TFjob？ #1466
• github.com/go-logr/zapr.(*zapLogger).Error #1444
• Display coverage % in GitHub actions list #1442
• Add Go test to CI #1436
• Podgroup is constantly created and deleted after tfjob is success or failure #1426
• Cut official release of 1.3.0 #1425
• Add "not maintained" notice to other operator repos #1423
• Fail to install tf-operator in minikube because of the version of kubectl/kustomize #1381
• Python SDK for Kubeflow Training Operator #1380
• Rename this repo #1348
• Universal Operator Phase III: Graduate operator to production grade #1318

v1.4.0-rc.0 release

Full Changelog

Features and improvements:

• Display coverage % in GitHub actions list #1442
• Add Go test to CI #1436

Fixed bugs:

• [bug] Missing init container in PyTorchJob #1482
• Fail to install tf-operator in minikube because of the version of kubectl/kustomize #1381

Closed issues:

• Restore KUBEFLOW_NAMESPACE options #1522
• Improve test coverage #1497
• swagger.json missing Pytorchjob.Spec.ElasticPolicy #1483
• PytorchJob DDP training will stop if I delete a worker pod #1478
• Write down e2e failure debug process #1467
• How can i add the Priorityclass to the TFjob？ #1466
• github.com/go-logr/zapr.(*zapLogger).Error #1444
• Podgroup is constantly created and deleted after tfjob is success or failure #1426
• Cut official release of 1.3.0 #1425
• Add "not maintained" notice to other operator repos #1423
• Python SDK for Kubeflow Training Operator #1380

Merged pull requests:

• Update manifests with latest image tag #1527 (johnugeorge)
• add option for mpi kubectl delivery #1525 (zw0610)
• restore option namespace in launch arguments #1524 (zw0610)
• remove unused scripts #1521 (zw0610)
• remove ChanYiLin from approvers #1513 (ChanYiLin)
• add StacktraceLevel for zapr #1512 (qiankunli)
• add unit tests for tensorflow controller #1511 (zw0610)
• add the example of MPIJob #1508 (hackerboy01)
• Added 2022 roadmap and migrated previous roadmap from kubeflow/common #1500 (terrytangyuan)
• Fix a typo in mpi controller log #1495 (LuBingtan)
• feat(pytorch): Add init container config to avoid DNS lookup failure #1493 (gaocegege)
• chore: Fix GitHub Actions script #1491 (tenzen-y)
• chore: Fix missspell in tfjob #1490 (tenzen-y)
• chore: Update OWNERS #1489 (gaocegege)
• Bump jinja2 from 2.10.1 to 2.11.3 in /py/kubeflow/tf_operator #1487 (dependabot[bot])
• fix comments for mpi-controller #1485 (hackerboy01)
• add expectation-related functions for other resources used in mpi-controller #1484 (zw0610)
• Add MPI job to README now that it's supported #1480 (terrytangyuan)
• add mpi doc #1477 (zw0610)
• Set Go version of base image to 1.17 #1476 (tenzen-y)
• update label for tf-controller #1474 (zw0610)
• Add Akuity to the list of adopters #1473 (terrytangyuan)
• Add PR template with doc checklist #1470 (andreyvelich)
• Add e2e failure debugging guidance #1469 (Jeffwan)
• chore: Add .gitattributes to ignore Jsonnet test code for linguist #1463 (terrytangyuan)
• Migrate additional examples from xgboost-operator #1461 (terrytangyuan)
• Minor edits to README.md #1460 (terrytangyuan)
• add mpi-operator(v1) to the unified operator #1457 (hackerboy01)
• fix tfjob status when enableDynamicWorker set true #1455 (zw0610)
• feat(pytorch): Support elastic training #1453 (gaocegege)
• fix: generate printer columns for job crds #1451 (henrysecond1)
• Fix README typo #1450 (davidxia)
• consistent naming for better readability #1449 (pramodrj07)
• Fix set scheduler error #1448 (qiankunli)
• Add CI to run the tests for Go #1440 (tenzen-y)
• fix: Add missing retrying package that failed the import #1439 (terrytangyuan)
• Generate a single swagger.json file for all frameworks #1437 (alembiewski)
• Update links and files with the new URL #1434 (andreyvelich)
• chore: update CHANGELOG.md #1432 (Jeffwan)
• Add acknowledgement section in README to credit all contributors #1422 (terrytangyuan)
• Add Cisco to Adopters List #1421 (andreyvelich)
• Add Python SDK for Kubeflow Training Operator #1420 (alembiewski)
• docs: Move myself to approvers #1419 (terrytangyuan)
• fix hyperlinks in the 'overview' section #1418 (pramodrj07)
• docs: Migrate adopters of all operators to this repo #1417 (terrytangyuan)
• Feature/support pytorchjob set queue of volcano #1415 (qiankunli)
• Bump controller-tools to 0.6.0 and enable GenerateEmbeddedObjectMeta #1409 (Jeffwan)
• Update scripts to generate sdk for all frameworks #1389 (Jeffwan)

v1.3.0

v1.3.0 (2021-10-03)

Full Changelog

Features

• Feature/support pytorchjob set queue of volcano (#1415, @qiankunli)
• Bump controller-tools to 0.6.0 and enable GenerateEmbeddedObjectMeta (#1409, @Jeffwan)
• add api-doc for all frameworks (#1370, @DeliangFan)
• Update training operator release process (#1347, @Jeffwan)
• add option to enable schemes (#1342, @zw0610)
• Don't use inline for runPolicy and register defaulter function (#1330, @Jeffwan)
• Use low-level controller and handlers in SetupWithManager (#1315, @Jeffwan)
• Reorganize controllers code structure (#1302, @Jeffwan)
• Unify code structure of training job api (#1300, @Jeffwan)
• Extract reusable codes to common utility (#1297, @Jeffwan)
• Add MXJob v1 api and controller (#1296, @Jeffwan)
• add tf reconciler to the unified operator (#1295, @zw0610)
• Add XGBoost controller (#1293, @Jeffwan)
• add pytorch API and controller (#1294, @zw0610)
• Generate tfjob 1.19.x compatible clientset (#1290, @Jeffwan)
• add XGBoostJob api (#1286, @zw0610)

Bug fixes

• fix hyperlinks in the 'overview' section (#1418, @pramodrj07)
• 2010: fix to expose correct monitoring port #1405 (deepak-muley)
• Fix 1399: added pod matching label in service selector #1404 (deepak-muley)
• fix: runPolicy validation error in the examples #1401 (Jeffwan)
• fix: volcano pod group creation issue (#1390, @Jeffwan)
• Fix 1340 prometheus counters (#1375, @deepak-muley)
• fix makefile to store crds in a separate folder (#1368, @deepak-muley)
• Fix copyright header for some files (#1371, @DeliangFan)
• fix incorrect torch env population (#1361, @Jeffwan)
• fix: Resolve scheme registration issue for defaulters (#1360, @Jeffwan)
• Fix XGBoost container name in log message (#1362, @andreyvelich)
• Fix postsubmit job using PULL_BASE_SHA (#1344, @Jeffwan)
• Fix all client request that needs contexts (#1292, @Jeffwan)

Misc

• Update manifests with latest image tag #1406 (johnugeorge)
• Add simple verification jobs (#1391, @Jeffwan)
• chore: Bump kubeflow/common version to 0.3.7 (#1388, @Jeffwan)
• chore(doc): Update README.md (#1387, @Jeffwan)
• Remove lagacy tf-operator from the codebase (#1378, @thunderboltsid)
• chore: Update manifest image tag (#1364, @Jeffwan)
• add example for mxnet and pytorch (#1373, @DeliangFan)
• docs: Update to use kubectl kustomize in instructions (#1356, @terrytangyuan)
• Clean up manifests and remove unused files (#1349, @Jeffwan)
• 1322: Modified manifests to use all-in-one training-operator (#1346, @deepak-muley)
• chore: Update changelog.md (#1339, @Jeffwan)
• Move root level docs to docs folder (#1338, @Jeffwan)
• Temporarily add Jeffwan@ to OWNERS (#1287, @Jeffwan)

Testing

• Update include dirs in prow config (#1374, @andreyvelich)
• Enable e2e test against universal operator (#1336, @Jeffwan)
• Use PULL_PULL_SHA for image (#1334, @PatrickXYS)

v1.3.0-rc.2

v1.3.0-rc.2 (2021-09-20)

Full Changelog

Closed issues:

• [bug] Unable to specify pod template metadata for TFJob #1403

Merged pull requests:

• Bump controller-tools to 0.6.0 and enable GenerateEmbeddedObjectMeta #1410 (Jeffwan)
• chore: Update image tags in manifests #1412 (Jeffwan)

v1.3.0-rc.1

v1.3.0-rc.1 (2021-09-15)

Full Changelog

Closed issues:

• [bug] Reconcilation fails when upgrading common to 0.3.6 #1394

Merged pull requests:

• Update manifests with latest image tag #1406 (johnugeorge)
• 2010: fix to expose correct monitoring port #1405 (deepak-muley)
• Fix 1399: added pod matching label in service selector #1404 (deepak-muley)
• fix: runPolicy validation error in the examples #1401 (Jeffwan)

v1.3.0-rc.0

v1.3.0-rc.0 (2021-08-31)

Full Changelog

Features

• add api-doc for all frameworks (#1370, @DeliangFan)
• Update training operator release process (#1347, @Jeffwan)
• add option to enable schemes (#1342, @zw0610)
• Don't use inline for runPolicy and register defaulter function (#1330, @Jeffwan)
• Use low-level controller and handlers in SetupWithManager (#1315, @Jeffwan)
• Reorganize controllers code structure (#1302, @Jeffwan)
• Unify code structure of training job api (#1300, @Jeffwan)
• Extract reusable codes to common utility (#1297, @Jeffwan)
• Add MXJob v1 api and controller (#1296, @Jeffwan)
• add tf reconciler to the unified operator (#1295, @zw0610)
• Add XGBoost controller (#1293, @Jeffwan)
• add pytorch API and controller (#1294, @zw0610)
• Generate tfjob 1.19.x compatible clientset (#1290, @Jeffwan)
• add XGBoostJob api (#1286, @zw0610)

Bug fixes

• fix: volcano pod group creation issue (#1390, @Jeffwan)
• Fix 1340 prometheus counters (#1375, @deepak-muley)
• fix makefile to store crds in a separate folder (#1368, @deepak-muley)
• Fix copyright header for some files (#1371, @DeliangFan)
• fix incorrect torch env population (#1361, @Jeffwan)
• fix: Resolve scheme registration issue for defaulters (#1360, @Jeffwan)
• Fix XGBoost container name in log message (#1362, @andreyvelich)
• Fix postsubmit job using PULL_BASE_SHA (#1344, @Jeffwan)
• Fix all client request that needs contexts (#1292, @Jeffwan)

Misc

• Add simple verification jobs (#1391, @Jeffwan)
• chore: Bump kubeflow/common version to 0.3.7 (#1388, @Jeffwan)
• chore(doc): Update README.md (#1387, @Jeffwan)
• Remove lagacy tf-operator from the codebase (#1378, @thunderboltsid)
• chore: Update manifest image tag (#1364, @Jeffwan)
• add example for mxnet and pytorch (#1373, @DeliangFan)
• docs: Update to use kubectl kustomize in instructions (#1356, @terrytangyuan)
• Clean up manifests and remove unused files (#1349, @Jeffwan)
• 1322: Modified manifests to use all-in-one training-operator (#1346, @deepak-muley)
• chore: Update changelog.md (#1339, @Jeffwan)
• Move root level docs to docs folder (#1338, @Jeffwan)
• Temporarily add Jeffwan@ to OWNERS (#1287, @Jeffwan)

Testing

• Update include dirs in prow config (#1374, @andreyvelich)
• Enable e2e test against universal operator (#1336, @Jeffwan)
• Use PULL_PULL_SHA for image (#1334, @PatrickXYS)

v1.3.0-alpha.3

v1.3.0 will be the first release version to support tensorflow, pytorch, mxnet and xgboost distributed training jobs.
More background can be found in design doc All-in-one Kubeflow Training Operator

Install Kubeflow training operator by running:

 kubectl apply -k "github.com/kubeflow/tf-operator.git/manifests/overlays/standalone?ref=v1.3.0-alpha.3"

require kubectl >= 1.21.x

v1.2.1

v1.2.1 (2021-08-27)

Full Changelog

Closed issues:

• volcano scheduler with customized queue #1377
• [chore] Delete the all-in-one branch #1372
• Fix scheme registration issue #1359
• Distributive Gloo PyTorchJob example doesn't work #1358
• Enable CI pipeline against v1.2-branch #1355
• Generate API Documentation for all frameworks #1341
• make test failed due to invalid crd schema #1324
• Cut 1.2.0 tag and release a stable version of master #1321
• Copyright header is not correctly generated #1309

Merged pull requests:

• fix(init): Fix crash problem when enabling gang scheduling #1384 (gaocegege)