clean structure #126
Merged
clean structure #126
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jakubczakon
reviewed
Jun 11, 2018
| @@ -3,7 +3,7 @@ | |||
| from attrdict import AttrDict | |||
| from deepsense import neptune | |||
|
|
|||
| from utils import read_params | |||
| from .utils import read_params | |||
|
|
|||
| ctx = neptune.Context() | |||
| params = read_params(ctx, fallback_file='neptune.yaml') | |||
There was a problem hiding this comment.
@taraspiotr not sure if this will work in python run ... under current structure (I suppose 'src/neptune.yaml' should).
| @@ -0,0 +1,94 @@ | |||
| import click | |||
There was a problem hiding this comment.
@taraspiotr I am on edge about naming and structure of stuff here because if we wanted to go with something django-like we would go with src -> app and main.py -> manage.py with pipeline_config.py -> settings.py . On the other hand we could simplify stuff and not have main.py fasade here at all (just explain in the instructions how to run it).
Frankly I don't mind the django-like version one bit. It would make it easy to create a API service around it (if we wanted to) while at the same time making it easily readable for every django/python dev.
@taraspiotr @apyskir thoughts?
src/pipeline_manager.py
Outdated
| create_annotations, generate_data_frame_chunks | ||
|
|
||
|
|
||
| class PipelineManager(): |
There was a problem hiding this comment.
@apyskir @taraspiotr I like that you use a facade here but I would rather have those methods as functions which are simply called when executing methods. That way we have both single responsibility and ease of use from main.py
src/steps/preprocessing/misc.py
Outdated
| @@ -3,7 +3,7 @@ | |||
| from sklearn.externals import joblib | |||
| from sklearn.feature_extraction import text | |||
|
|
|||
| from steps.base import BaseTransformer | |||
| from ...steps.base import BaseTransformer | |||
There was a problem hiding this comment.
@taraspiotr I didn't know about the ... concept.
Now that I did a quick search this came up:
However one of the guys running the workshop said that we should use no more than two dots and Guido van Rossum would personally chop off the fingers of those who use triple dot notation for relative imports. I always recall this when it comes to triple dot notation so I avoid to use it.
So I'd rather not use it and think about what can be done about it. Sklearn deals with similar problems by putting stuff in __init__.py https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/ensemble/__init__.py
added 2 commits
June 12, 2018 16:59
jakubczakon
added a commit
that referenced
this pull request
Jun 15, 2018
* initial restructure * clean structure (#126) * clean structure * correct readme * further cleaning * Dev apply transformer (#131) * clean structure * correct readme * further cleaning * resizer docstring * couple docstrings * make apply transformer, memory cache * fixes * postprocessing docstrings * fixes in PR * Dev repo cleanup (#132) * cleanup * remove src. * Dev clean tta (#134) * added resize padding, refactored inference pipelines * refactored piepliens * added color shift augmentation * reduced caching to just mask_resize * updated config * Dev-repo_cleanup models and losses docstrings (#135) * models and losses docstrings * small fixes in docstrings * resolve conflicts in with TTA PR (#137)
jakubczakon
added a commit
that referenced
this pull request
Jun 19, 2018
* added gmean tta, experimented with thresholding (#125) * Dev repo cleanup (#138) * initial restructure * clean structure (#126) * clean structure * correct readme * further cleaning * Dev apply transformer (#131) * clean structure * correct readme * further cleaning * resizer docstring * couple docstrings * make apply transformer, memory cache * fixes * postprocessing docstrings * fixes in PR * Dev repo cleanup (#132) * cleanup * remove src. * Dev clean tta (#134) * added resize padding, refactored inference pipelines * refactored piepliens * added color shift augmentation * reduced caching to just mask_resize * updated config * Dev-repo_cleanup models and losses docstrings (#135) * models and losses docstrings * small fixes in docstrings * resolve conflicts in with TTA PR (#137) * refactor in stream mode (#139) * hot fix of mask_postprocessing in tta with new make transformer * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * local * Update README.md * Update README.md * Update README.md * Update README.md * Dev preparation path fix (#140) * local * cleaned up paths in the masks and metadata generation * dropped debug stuff * Dev non trainable transformer flag (#141) * local * added is_trainable flag to models
jakubczakon
pushed a commit
that referenced
this pull request
Jun 21, 2018
* initial restructure * thresholds on unet output * added gmean tta, experimented with thresholding (#125) * feature exractor and lightgbm * pipeline is running ok * tmp commit * lgbm ready for tests * tmp * faster nms and feature extraction * small fix * cleaning * Dev repo cleanup (#138) * initial restructure * clean structure (#126) * clean structure * correct readme * further cleaning * Dev apply transformer (#131) * clean structure * correct readme * further cleaning * resizer docstring * couple docstrings * make apply transformer, memory cache * fixes * postprocessing docstrings * fixes in PR * Dev repo cleanup (#132) * cleanup * remove src. * Dev clean tta (#134) * added resize padding, refactored inference pipelines * refactored piepliens * added color shift augmentation * reduced caching to just mask_resize * updated config * Dev-repo_cleanup models and losses docstrings (#135) * models and losses docstrings * small fixes in docstrings * resolve conflicts in with TTA PR (#137) * refactor in stream mode (#139) * hot fix of mask_postprocessing in tta with new make transformer * finishing merge * finishing merge v2 * finishing merge v3 * finishing merge v4 * tmp commit * lgbm train and evaluate pipelines run correctly * something is not yes * fix * working lgbm training with ugly train_mode=True * back to pipelines.py * small fix * preparing PR * preparing PR v2 * preparing PR v2 * fix * fix_2 * fix_3 * fix_4
jakubczakon
added a commit
that referenced
this pull request
Jun 21, 2018
* initial restructure * thresholds on unet output * added gmean tta, experimented with thresholding (#125) * feature exractor and lightgbm * pipeline is running ok * tmp commit * lgbm ready for tests * tmp * faster nms and feature extraction * small fix * cleaning * Dev repo cleanup (#138) * initial restructure * clean structure (#126) * clean structure * correct readme * further cleaning * Dev apply transformer (#131) * clean structure * correct readme * further cleaning * resizer docstring * couple docstrings * make apply transformer, memory cache * fixes * postprocessing docstrings * fixes in PR * Dev repo cleanup (#132) * cleanup * remove src. * Dev clean tta (#134) * added resize padding, refactored inference pipelines * refactored piepliens * added color shift augmentation * reduced caching to just mask_resize * updated config * Dev-repo_cleanup models and losses docstrings (#135) * models and losses docstrings * small fixes in docstrings * resolve conflicts in with TTA PR (#137) * refactor in stream mode (#139) * hot fix of mask_postprocessing in tta with new make transformer * finishing merge * finishing merge v2 * finishing merge v3 * finishing merge v4 * tmp commit * lgbm train and evaluate pipelines run correctly * something is not yes * fix * working lgbm training with ugly train_mode=True * back to pipelines.py * small fix * preparing PR * preparing PR v2 * preparing PR v2 * fix * fix_2 * fix_3 * fix_4
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.