Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cifar10 transfer learning example #2029

Closed
wants to merge 1,518 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1518 commits
Select commit Hold shift + click to select a range
36d8e8a
Internal changes.
tfx-copybara Apr 23, 2020
43792f4
Updates template notebook with the new release.
Apr 24, 2020
22a305d
Removed `python-snappy` from `[all]` extra dependency list.
Apr 24, 2020
ddfdfdc
test only change.
tfx-copybara Apr 24, 2020
021f7cc
Implementation of TFJS Rewriter.
tfx-copybara Apr 24, 2020
89f37ba
Implement support for execution parameters and optional inputs in Pyt…
tfx-copybara Apr 24, 2020
7512789
Unify Pusher behavior.
chongkong Apr 24, 2020
9158889
Automated rollback of commit 307bda13c226534cfb649f4155421cb91b932295
tfx-copybara Apr 24, 2020
783c979
internal change
brills Apr 24, 2020
997440a
Move Python function component code to under `tfx.dsl.component.exper…
tfx-copybara Apr 27, 2020
1c74e0a
Automated rollback of commit a0a4da2feb0b822a41055b825fafe2663a4f1a8e
tfx-copybara Apr 27, 2020
34a286d
test only
tfx-copybara Apr 27, 2020
1f58af3
PR #1667: Refreshes the contributing.md.
Apr 28, 2020
d128d50
Adds pylintrc and github action for linting and testing.
Apr 28, 2020
5c43b78
test only
tfx-copybara Apr 28, 2020
0b13e84
internal clean up
tfx-copybara Apr 28, 2020
a31d5c1
internal
tfx-copybara Apr 28, 2020
86ac08c
Guard optional import error not to propagate to outer scope.
Apr 28, 2020
0431c18
Cleanup CAIP tutorial and add Dataflow quota note.
rcrowe-google Apr 28, 2020
65e5d2e
Lower accuracy threshold to 0.6.
Apr 29, 2020
6f33907
test only
ucdmkt Apr 29, 2020
b2db929
Do not use requirements.txt when specifying TFX dependency with Beam.
tfx-copybara Apr 29, 2020
f314866
Uses > Use
rcrowe-google Apr 29, 2020
0767d2e
Add execution properties to ExecutionOutput proto
tfx-copybara Apr 29, 2020
4827c3c
internal
tfx-copybara Apr 30, 2020
993b055
Added the TemplatedExecutorContainerSpec class that supports placehol…
tfx-copybara Apr 30, 2020
6c14fac
no op
tfx-copybara Apr 30, 2020
e2739f3
Add KFP resource configuration discussion.
rcrowe-google May 1, 2020
fef0e5a
Adds license of the package `packaging`.
May 1, 2020
5d9896d
Update TFX native keras tests
tfx-copybara May 1, 2020
4e34267
Update taxi dataset to filter out examples with no label (tip).
embr May 1, 2020
b62e4be
Support regenerating datasets in user-specified directory.
embr May 1, 2020
8ea2dce
Fixes unintentional dependencies overriding in KFP container entrypoint.
tfx-copybara May 1, 2020
406b426
Fix several PTransform typehints related to instance key type (should…
tfx-copybara May 3, 2020
b507bcc
Fix component instance names to match pipeline.py
kweinmeister May 3, 2020
b33c64d
Added note about permissions configuration for GCR
kweinmeister May 3, 2020
c700c10
Fix link to testing conventions
domedagen May 5, 2020
4a9db3e
Add space to "This project followsGoogle's"
domedagen May 5, 2020
28d2e18
CONTRIBUTING.md: Note on Python 3.8 and above
domedagen May 5, 2020
d5dfc3f
CONTRIBUTING.md: Note on protoc
domedagen May 5, 2020
3405423
Enable bigquery example gen to use new (experimental) ReadFromBigquer…
May 5, 2020
5267848
Update CONTRIBUTING.md
domedagen May 6, 2020
ecb5933
Adds license of new dependent packages.
May 6, 2020
bf7ff07
Added the experimental function "create_container_component" that sim…
tfx-copybara May 6, 2020
81252ef
Fixes some typehint error.
tfx-copybara May 6, 2020
e2e4833
Fail-fast AI Platform Prediction model deployment error
ucdmkt May 6, 2020
330198b
Merge pull request #1730 from domedagen:patch-4
tfx-copybara May 6, 2020
0cb0e8c
Merge pull request #1729 from domedagen:patch-3
tfx-copybara May 6, 2020
8baa9cf
Merge pull request #1728 from domedagen/patch-1
1025KB May 6, 2020
aaed692
Merge pull request #1741 from tensorflow/test_310193590
1025KB May 6, 2020
038bd73
Remove Cifar10 example (keep mnist as the example for image)
tfx-copybara May 6, 2020
7fa2cc3
Changes custom config for the Do() functions of Trainer and Pusher to…
tfx-copybara May 6, 2020
f561115
Remove batched keys. Instead use the same keys for batched and unbatc…
paulgc May 7, 2020
08d0c9c
Upgraded pyarrow dep to 0.16.
brills May 7, 2020
08d8484
fix airflow pipeline config kwargs
casassg May 7, 2020
a18997e
Pin Tensorflow version to 2.1 temporarily
tfx-copybara May 8, 2020
858c160
Internal change
May 8, 2020
dba8c95
Call out requirement of --temp_location for BQ example gen.
May 8, 2020
382fde9
Fix type hint of file_pattern in _input_fn
tfx-copybara May 8, 2020
fda18a1
test only
tfx-copybara May 8, 2020
f478bb2
Fix census notebook
tfx-copybara May 9, 2020
a4df752
Depends on frozendict.
tfx-copybara May 9, 2020
24362f3
Fix census notebook, pin tf to 2.1
tfx-copybara May 9, 2020
ac38000
Merge pull request #1753 from casassg:gcasassaez/fix_airflow_pipeline…
tfx-copybara May 11, 2020
671d177
Fixes example code errors in mlmd guide.
tfx-copybara May 11, 2020
2930bda
Fixing bad anchor tags and links in TFDV guide
rcrowe-google May 11, 2020
4ad3b7b
Adds a github action for linting and testing.
May 12, 2020
453f188
fix link
krystollia May 12, 2020
d971e29
Add InfraValidator to the rest of the guides.
chongkong May 12, 2020
e65794f
Merge pull request #1723 from kweinmeister:master
tfx-copybara May 12, 2020
16a2bc0
Fixing "training-serving" anchor link
rcrowe-google May 13, 2020
adc1512
CONTRIBUTING.md Fix broken link and add space
domedagen May 13, 2020
f82e23f
no op
tfx-copybara May 13, 2020
c8379c5
Enhanced add_upstream_node() and add_downstream_node() to support tas…
tfx-copybara May 13, 2020
c29ae10
Cleanup notebook document for taxi template on Beam orchestrator.
May 14, 2020
ed4d5f2
Add AUC benchmark for TFMA v2
embr May 14, 2020
578ea49
Made ComponentSpec().inputs and .outputs behave more like real dictio…
tfx-copybara May 14, 2020
14c17b4
Adds license of new dependent packages.
tfx-copybara May 14, 2020
60150e4
Refactoring: Converted the BaseNode class attributes to constructor p…
tfx-copybara May 14, 2020
7ca7b6b
Use absl logging functions to prevent double logging.
May 14, 2020
484dfde
Changes the default k8s launcher timeout to 300s.
tfx-copybara May 14, 2020
66b9407
PR #1731: Add support for BigQuery arrays in BigQueryExampleGen and u…
tfx-copybara May 14, 2020
887101d
Modify "extra_package" to "extra_packages" because Beam will automati…
tfx-copybara May 14, 2020
e2f1c6d
Allows user to specify CAIP runtime version in cloud Pusher.
tfx-copybara May 14, 2020
6ac928d
Update package versions for the census notebook.
May 15, 2020
a0bb587
Introduces machine_type argument for beam pipeline args to mitigate I…
May 15, 2020
26430c0
Fixes typo and other minor issue.
tfx-copybara May 15, 2020
e290c64
internal change
tfx-copybara May 15, 2020
2256dfa
Undo pyzmq version pinning.
chongkong May 15, 2020
904f791
Polished explanation for worker_type in beam_pipeline_args
May 15, 2020
1447c90
Merge pull request #1776 from krystollia:patch-2
tfx-copybara May 15, 2020
4102491
Fix broken link (bit.ly/fairness-indicators-guidance) in the doc.
chongkong May 15, 2020
67c8a63
Comment change for pipeline.proto.
tfx-copybara May 16, 2020
7573858
Merge pull request #1792 from domedagen:patch-1
tfx-copybara May 16, 2020
875632f
Fix list comprehension for bucket features in tfx template.
May 18, 2020
239f73a
Change the e2e tests to use out-of-graph metrics other than in-graph …
tfx-copybara May 18, 2020
8f11eec
Fix a bug where the tax_pipeline_beam is using an nonexistent default…
tfx-copybara May 18, 2020
d1a443d
Simplify AIP Prediction Pusher polling loop
ucdmkt May 18, 2020
803f177
Switch tfx examples to use bucketize with always_return_num_quantiles…
tfx-copybara May 18, 2020
f7daa8a
Refactoring ExampleGen driver
tfx-copybara May 18, 2020
13301c5
TFXIO implementation for for csv.
tfx-copybara May 18, 2020
3e70ae9
Automated rollback of commit eeca90aa21c254a786d6d9b8de32a50d2ef8d8b3
May 19, 2020
0819afa
Change the examples to use out-of-graph metrics other than in-graph o…
May 19, 2020
f0e587d
Fixed python2 compatibility in argument resolving for TemplatedExecut…
tfx-copybara May 19, 2020
46a4116
Polished template notebook document for Beam orchestrator.
May 19, 2020
3e47990
Fix the bug where tfma is using a nonexistent default metric.
tfx-copybara May 20, 2020
87ce0df
Fixes incompatibility in CAIP model runtime version and associated LRO.
tfx-copybara May 20, 2020
6dfefb7
Syntax Error.
MarkDaoust May 20, 2020
a80a3dc
Redirect stdout and stderr for temp package build, to minimize noise …
May 20, 2020
b959c37
Refactor Trainer for upcoming Tuner component
tfx-copybara May 20, 2020
b97311c
Change fixed import path in the template handler.
May 21, 2020
76d92ce
Change get_default_pod_labels() to a public function.
May 21, 2020
f2d0240
Update Taxi keras example to use a dataset without label transformation
tfx-copybara May 21, 2020
ad15cbd
Add Tuner component based on RFC[1]
tfx-copybara May 21, 2020
5e53ddd
noop
tfx-copybara May 22, 2020
4f79fdd
Updated TF2 language and recommendations
rcrowe-google May 24, 2020
e59c63f
Add a WIP readme for tuner
tfx-copybara May 26, 2020
99bb6d4
Added the container-based sample pipeline (download, filter, print)
tfx-copybara May 26, 2020
36e65e4
Added the scripts/run_component.py script.
tfx-copybara May 26, 2020
e21361a
Add missing %%skip_for_export annotations to notebook examples.
tfx-copybara May 27, 2020
893ffd9
Raising visibility of RFCs to encourage community participation.
rcrowe-google May 27, 2020
60c8c43
Updating TFDV install
rcrowe-google May 27, 2020
d4fc3f1
Adds license of new dependent packages.
May 27, 2020
756df8a
Use 1 Beam worker process by default in exported TFX notebook pipeline.
tfx-copybara May 27, 2020
e120133
Use mysql in the test of airflow dag runner.
May 27, 2020
5873be6
Automated rollback of commit 1d6b676b569e05837c9ca9b849a62284ffecb828
tfx-copybara May 27, 2020
ccd6dce
internal
tfx-copybara May 28, 2020
58b74cc
Add StructuralRuntimeParameter. Also move `output_key` out of `Artifa…
tfx-copybara May 28, 2020
0fb37a8
Depends on `pyyaml>=3.12,<6`. This is to make kubernetes 10.1.* happy…
May 28, 2020
367e441
Moving RFCs to opensource_only and updating copybara
rcrowe-google May 28, 2020
f20ee8e
Depends on `apache-beam[gcp]>=2.21,<3`. This should include some perf…
May 28, 2020
e1dc680
Use automatically generated GCS bucket for CAIP pipelines.
May 29, 2020
4b04843
Change the default template back to use a estimator based model due t…
tfx-copybara May 29, 2020
e5ea558
Update keras model in Taxi template to use a dataset without label tr…
May 29, 2020
94678e2
Clarifying restart runtime instructions and adding check
rcrowe-google May 29, 2020
bc32b4b
Updating pip before installing TFX
rcrowe-google May 29, 2020
4742256
Partially rollback the part for import cleanup in cl/313680840. I mis…
May 31, 2020
794000e
Changes the template default model to Keras again in favor of PR#1869.
tfx-copybara Jun 1, 2020
9ef4d20
Try install tfdv too and see if that fixes the kokoro build.
yashk2810 Jun 1, 2020
8639662
Explanation of Colab test for Pip upgrade
rcrowe-google Jun 1, 2020
6fba649
Added support for container components to BeamDagRunner.
tfx-copybara Jun 1, 2020
86eba2f
Internal change
yashk2810 Jun 1, 2020
0e37935
Added simple artifact types to types/experimental folder.
tfx-copybara Jun 2, 2020
e993c54
Fix license location for https://raw.githubusercontent.com/jquast/wcw…
Jun 2, 2020
6477de9
Automated rollback of commit d3b97a5bab4e117ef74e99a4a859888326433907
yashk2810 Jun 2, 2020
2a3a404
Improve URI construction in some tests.
tfx-copybara Jun 2, 2020
62f092c
Remove usage of frozendict before release to minimize dependency.
Jun 2, 2020
884d50c
cleanup
ucdmkt Jun 2, 2020
4e07cf6
Fixed the dsl/component/experimental/decorator_test.
tfx-copybara Jun 2, 2020
6df1dda
Adding TensorBoard callback to template Keras taxi model
rcrowe-google Jun 3, 2020
abf2af9
Internal change
Jun 3, 2020
bd811f7
Make sure dockerfile_test is not sensitive to version string change.
Jun 3, 2020
6e7af0e
Adding BulkInferrer to the left nav with the other components
rcrowe-google Jun 3, 2020
dac86c8
Update README.md
pingsutw Jun 3, 2020
5d0361d
Making output_artifacts in executor outputs Dict[Text, List[Artifacts…
tfx-copybara Jun 3, 2020
69c4998
TFX 0.22.0 Release
dhruvesh09 Jun 3, 2020
2120154
test only change
tfx-copybara Jun 3, 2020
cc4771c
Allow `instance_name` parameter in function component constructor.
tfx-copybara Jun 3, 2020
e00aec3
internal change
tfx-copybara Jun 3, 2020
49e8336
Fix typo in CSVExampleGen executor.
tfx-copybara Jun 4, 2020
38a55e7
Change model validator to evaluator in comments.
Jun 4, 2020
4779821
Enable float execution properties.
tfx-copybara Jun 4, 2020
6303630
Add Tuner component to Iris e2e Example
tfx-copybara Jun 4, 2020
c6f8d7f
Fix io_utils.copy_dir issue and update unit test
tfx-copybara Jun 4, 2020
8cc93ee
Update components __init__ file to include tuner
tfx-copybara Jun 4, 2020
c5ad4e0
Changed the encoding of the Integer and Float artifacts to be more po…
tfx-copybara Jun 5, 2020
1770f11
Refresh release log for 0.22.0:
Jun 5, 2020
d467076
Documentation - Creating container-based components
tfx-copybara Jun 5, 2020
8dfd8c0
Added pipeline based on create_container_component to Kubeflow e2e te…
tfx-copybara Jun 5, 2020
784854f
Adding uDSL RFC to current list
rcrowe-google Jun 6, 2020
5b4ea70
internal change
tfx-copybara Jun 8, 2020
02c600f
noop
tfx-copybara Jun 9, 2020
e4896f3
Adds TODO for the removal of `machine_type` parameter and other tempo…
Jun 9, 2020
5113b06
Use import instead of exec() in setup.py
chongkong Jun 9, 2020
b104242
Fixes a typo in ExecutorContainerSpec doc string.
tfx-copybara Jun 9, 2020
8b1e1c5
Clean up flag TFXIO_HAS_TELEMETRY.
brills Jun 9, 2020
c42947a
Cleaned up flag TFDV_ACCEPT_RECORD_BATCH
brills Jun 9, 2020
45bb6fd
add feature based partition
tinally Jun 10, 2020
e779034
no op
tfx-copybara Jun 10, 2020
4a44043
testing only
tfx-copybara Jun 10, 2020
5329de2
Improves the error handling in extensions.google_caip.trainer.runner.…
tfx-copybara Jun 10, 2020
1ff8d14
internal
tfx-copybara Jun 10, 2020
399fa56
Updates TFX docs to describe custom Python function components.
tfx-copybara Jun 10, 2020
a9f41df
testing only
tfx-copybara Jun 10, 2020
d3f2cc5
Updates TFX CLI documentation to describe command arguments and restr…
tfx-copybara Jun 10, 2020
6ab6148
Fix for minor typo in index.md in TFX user guide
tfx-copybara Jun 10, 2020
3292e72
Adds documentation for TFX pipeline and component core concepts.
tfx-copybara Jun 10, 2020
af9420b
Adds documentation for building a TFX pipeline.
tfx-copybara Jun 10, 2020
9ac6778
Updates TFX docs to better describe TFX pipelines and add new docs to…
tfx-copybara Jun 10, 2020
652efe7
Use existing API to write statistics
paulgc Jun 10, 2020
03520bf
add unit test for feature based partition
tinally Jun 11, 2020
9401b2e
Related to #1754: relax the restriction that output uris must be newl…
tfx-copybara Jun 11, 2020
4a049f5
internal change
tfx-copybara Jun 11, 2020
f1274dc
fixed code structure and cleanup
tinally Jun 11, 2020
efdbfa9
Syncing exercise and solution for Airflow workshop, and adding Tensor…
rcrowe-google Jun 11, 2020
9908f01
add splittig_method_config and raise error for float_list features
tinally Jun 11, 2020
bbd598d
check feature name instead of the config
tinally Jun 11, 2020
b1cdd06
Fixes broken links in the TFX docs.
tfx-copybara Jun 11, 2020
ee409d7
noop
tfx-copybara Jun 12, 2020
891cb0b
Don't delete internal TODOs any more while copying template codes.
Jun 12, 2020
6c46a50
Make ModelBlessing optional in Pusher.
chongkong Jun 12, 2020
5c36eeb
Update README.md
pingsutw Jun 12, 2020
332c21f
Fixing 404 in standard components link
rcrowe-google Jun 12, 2020
4f86c5c
Update BulkInferrer doc and link.
tfx-copybara Jun 12, 2020
e7e12ac
Update bulkinferrer.md
meixinzhang23 Jun 12, 2020
1ff2fc4
merge with upstream
meixinzhang23 Jun 12, 2020
6d71ce5
Updating Keras Colab to TFX 0.22.0
rcrowe-google Jun 13, 2020
2cd1352
Moved BigQueryExampleGen to `tfx.extensions.google_cloud_big_query`. …
Jun 14, 2020
8dd3e90
Merge pull request #1918 from pingsutw:pingsutw-patch-1
tfx-copybara Jun 15, 2020
027a646
Use TestDataflowRunner and a 20 min timeout for dataflow jobs.
Jun 15, 2020
b65c6b6
Automated rollback of commit 0c307ab98a299475e16bf118eea939a38c76bef1
Jun 15, 2020
97414a5
Creates resolve_deps setuptools command.
chongkong Jun 15, 2020
eea8103
Remove redundant word
meixinzhang23 Jun 15, 2020
e61a175
Switch the default input location for CsvExampleGen to a GCS path ins…
tfx-copybara Jun 15, 2020
d5c676e
Change all links to official tensorflow guide
meixinzhang23 Jun 15, 2020
e4a846f
added tests for invalid feature name and float_list features
tinally Jun 15, 2020
cea8496
check error message
tinally Jun 15, 2020
7f0cc8f
use _exec_properties
tinally Jun 15, 2020
05a396f
Small modification to DSL.
tfx-copybara Jun 15, 2020
d591517
Merge pull request #1973 from tinally:master
tfx-copybara Jun 15, 2020
fd049cb
Set proper number of steps for keras chicago taxi model.
Jun 16, 2020
cb22780
Internal change
Jun 16, 2020
5b0d35a
Added DSL IR compiler that encodes a TFX pipeline into a DSL proto.
tfx-copybara Jun 16, 2020
7919e09
Updating Estimator Colab for TFX 0.22.0
rcrowe-google Jun 16, 2020
1b008bf
Internal change
tfx-copybara Jun 16, 2020
22cc86d
Clarify documentation regarding incompatible dependency version errors.
tfx-copybara Jun 16, 2020
b548fa8
Moved BigQueryExampleGen to `tfx.extensions.google_cloud_big_query`.
Jun 16, 2020
35b5169
Small fix
tfx-copybara Jun 16, 2020
b5a780b
Fixing imports on function component example code
rcrowe-google Jun 17, 2020
f469826
Merge pull request #1989 from meixinzhang:master
tfx-copybara Jun 18, 2020
99fc258
Add analysis paths metric to the Transform executor.
tfx-copybara Jun 18, 2020
e9ce77a
Added Tuner guide
tfx-copybara Jun 18, 2020
81c5c52
Remove Tuner from custom_components/ as it's supported under components/
tfx-copybara Jun 18, 2020
73eb352
TFX cleanup after beam supports multi-process directly with flag
tfx-copybara Jun 19, 2020
cb12bcb
Fixes wrong update of KFP pod labels.
Jun 19, 2020
8cd01cc
Moving Tuner RFC to approved
rcrowe-google Jun 19, 2020
b34040e
Merge pull request #2026 from tensorflow/test_317326777
1025KB Jun 19, 2020
e697704
Update TFX examples
tfx-copybara Jun 19, 2020
7652911
cifar10 transfer learning example
Deanplayerljx Jun 21, 2020
c14b6df
address concerns
Deanplayerljx Jun 26, 2020
dfe9c9f
address concerns and add finetuning
Deanplayerljx Jul 7, 2020
37aefdc
add tiny dataset(100 image)
Deanplayerljx Jul 8, 2020
8370937
add test for pipeline
Deanplayerljx Jul 8, 2020
91ddc1c
add metadata writer
Deanplayerljx Jul 8, 2020
67c3ff5
update test
Deanplayerljx Jul 8, 2020
a2c024c
fix syntax in test
Deanplayerljx Jul 8, 2020
360a0bb
address concerns for the pipeline
Deanplayerljx Jul 9, 2020
517a180
add label map and clean up comments
Deanplayerljx Jul 9, 2020
2e28136
update readme
Deanplayerljx Jul 9, 2020
bc4ba13
add data augmentation, needs debugging evaluator
Deanplayerljx Jul 16, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
138 changes: 138 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# LINT.IfChange
# Byte-compiled / optimized / DLL files
__pycache__/
**/*.py[cod]
**/*$py.class

# C extensions
**/*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don’t work, or not
# install all needed dependencies.
#Pipfile.lock

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# Intellij project settings
.idea

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pb2.py files
**/*_pb2.py
# LINT.ThenChange(.gitignore)

# Github integration settings
.github

# .dockerignore and Dockerfile themselves are not needed by docker build.
.dockerignore
tfx/tools/docker/Dockerfile*
67 changes: 67 additions & 0 deletions .github/workflows/ci-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Github action definition for ci-test with PRs.

name: tfx-ci-test
on:
pull_request:
branches: [ master ]

jobs:
build:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: Get Changed Files
id: changed_files
uses: trilom/[email protected]
with:
fileOutput: ' '
- name: Select files to check
run: |
# Filter out non-python files.
cat $HOME/files_added.txt $HOME/files_modified.txt | tr ' ' '\n' | grep '\.py$' > py_files.txt || true
# Filter out non-test python files and e2e or integration tests.
cat py_files.txt | grep '_test\.py$' | grep -v _e2e_ | grep -v integration | grep -v 'examples/' > py_test_files.txt || true
# Select proto files.
cat $HOME/files_added.txt $HOME/files_modified.txt | tr ' ' '\n' | grep '\.proto$' > proto_files.txt || true

- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
env:
PROTO_VERSION: 3.11.4
PROTOC: bin/protoc
run: |
python -m pip install --upgrade pip
curl -sSOL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTO_VERSION}/protoc-${PROTO_VERSION}-linux-x86_64.zip
unzip protoc-${PROTO_VERSION}-linux-x86_64.zip
pip install -e .[all]

- name: Run unit tests
shell: bash
run: |
[ ! -s "py_test_files.txt" ] || cat py_test_files.txt | xargs -I {} python {}

- name: Lint with protolint
env:
PROTOLINT_VERSION: 0.25.1
shell: bash
run: |
curl -sSOL https://github.com/yoheimuta/protolint/releases/download/v${PROTOLINT_VERSION}/protolint_${PROTOLINT_VERSION}_Linux_x86_64.tar.gz
tar zxf protolint_${PROTOLINT_VERSION}_Linux_x86_64.tar.gz
echo "[NOTE] This linter is currently EXPERIMENTAL.======================================="
echo "Please contact reviewers for existing lint errors or false negative errors."
echo "===================================================================================="
[ ! -s "proto_files.txt" ] || cat proto_files.txt | xargs -I {} ./protolint {}

- name: Lint with pylint
shell: bash
run: |
pip install pylint
echo "[NOTE] This linter is currently EXPERIMENTAL.======================================="
echo "Please contact reviewers for existing lint errors or false negative errors."
echo "Feel free to send PRs for pylintrc in the root directory of the repository if needed."
echo "===================================================================================="
[ ! -s "py_files.txt" ] || pylint $(cat py_files.txt | tr '\n' ' ')
131 changes: 131 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# LINT.IfChange
# Byte-compiled / optimized / DLL files
__pycache__/
**/*.py[cod]
**/*$py.class

# C extensions
**/*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don’t work, or not
# install all needed dependencies.
#Pipfile.lock

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# Intellij project settings
.idea

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pb2.py files
**/*_pb2.py
# LINT.ThenChange(.dockerignore)
Loading