microsoft · yqwang-ms · Nov 23, 2018 · Nov 20, 2018 · Nov 20, 2018 · Nov 20, 2018
diff --git a/README.md b/README.md
@@ -58,11 +58,8 @@ A Framework represents an application with a set of Tasks:
 
 ## Quick Start
 1. [Build](build/frameworkcontroller)
-2. [Run Example](example/run/frameworkcontroller)
-3. [Config Usage](pkg/apis/frameworkcontroller/v1/config.go)
-4. [Config Example](example/config)
-5. [Framework Usage](pkg/apis/frameworkcontroller/v1/types.go)
-6. [Framework Example](example/framework)
+2. [Run Example](example/run/frameworkcontroller.md)
+3. [Framework Example](example/framework)
 
 ## Doc
 1. [User Manual](doc/user-manual.md)
@@ -76,13 +73,13 @@ A specialized wrapper can be built on top of FrameworkController to optimize for
 * [NNI Controller Wrapper](https://github.com/Microsoft/nni)(Developing): A wrapper client optimized for AutoML applications
 
 ## Official Image
-[FrameworkController DockerHub](https://hub.docker.com/u/frameworkcontroller)
+* [DockerHub](https://hub.docker.com/u/frameworkcontroller)
 
 ## Related Project
 * [YARN FrameworkLauncher](https://github.com/Microsoft/pai/blob/master/subprojects/frameworklauncher/yarn): Similar offering natively supports [Apache YARN](http://hadoop.apache.org)
 
 ## Contributing
-This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+This project welcomes contributions and suggestions. Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
 the rights to use your contribution. For details, visit https://cla.microsoft.com.
 

diff --git a/bin/frameworkbarrier/start.sh b/bin/frameworkbarrier/start.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# MIT License
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+BASH_DIR=$(cd $(dirname ${BASH_SOURCE}) && pwd)
+
+cd ${BASH_DIR}
+
+./frameworkbarrier
+
+MNT_DIR=/mnt/frameworkbarrier
+
+mkdir -p ${MNT_DIR}
+
+cp -r ./framework.json ${MNT_DIR}
+cp -r ./injector.sh ${MNT_DIR}
+
+echo Succeeded to copy current Framework helper files into ${MNT_DIR}:
+cd ${MNT_DIR} && ls -lR .
diff --git a/build/frameworkbarrier/Dockerfile b/build/frameworkbarrier/Dockerfile
@@ -0,0 +1,43 @@
+# MIT License
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE
+
+FROM golang:alpine as builder
+
+ENV PROJECT_DIR=${GOPATH}/src/github.com/microsoft/frameworkcontroller
+ENV INSTALL_DIR=/opt/frameworkcontroller/frameworkbarrier
+
+RUN apk update && apk add --no-cache bash && \
+  mkdir -p ${PROJECT_DIR} ${INSTALL_DIR}
+COPY . ${PROJECT_DIR}
+RUN ${PROJECT_DIR}/build/frameworkbarrier/go-build.sh && \
+  mv ${PROJECT_DIR}/dist/frameworkbarrier/* ${INSTALL_DIR}
+
+
+FROM alpine:latest
+
+ENV INSTALL_DIR=/opt/frameworkcontroller/frameworkbarrier
+
+RUN apk update && apk add --no-cache bash
+COPY --from=builder ${INSTALL_DIR} ${INSTALL_DIR}
+WORKDIR ${INSTALL_DIR}
+
+ENTRYPOINT ["./start.sh"]
diff --git a/build/frameworkbarrier/docker-build.sh b/build/frameworkbarrier/docker-build.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# MIT License
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+BASH_DIR=$(cd $(dirname ${BASH_SOURCE}) && pwd)
+PROJECT_DIR=${BASH_DIR}/../..
+IMAGE_NAME=frameworkbarrier
+
+cd ${PROJECT_DIR}
+
+docker build -t ${IMAGE_NAME} -f ${BASH_DIR}/Dockerfile .
+
+echo Succeeded to build docker image ${IMAGE_NAME}
diff --git a/build/frameworkbarrier/go-build.sh b/build/frameworkbarrier/go-build.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# MIT License
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+BASH_DIR=$(cd $(dirname ${BASH_SOURCE}) && pwd)
+# Ensure ${PROJECT_DIR} is ${GOPATH}/src/github.com/microsoft/frameworkcontroller
+PROJECT_DIR=${BASH_DIR}/../..
+DIST_DIR=${PROJECT_DIR}/dist/frameworkbarrier
+
+cd ${PROJECT_DIR}
+
+rm -rf ${DIST_DIR}
+mkdir -p ${DIST_DIR}
+
+go build -o ${DIST_DIR}/frameworkbarrier cmd/frameworkbarrier/*
+chmod a+x ${DIST_DIR}/frameworkbarrier
+cp -r bin/frameworkbarrier/* ${DIST_DIR}
+
+echo Succeeded to build binary distribution into ${DIST_DIR}:
+cd ${DIST_DIR} && ls -lR .
diff --git a/build/frameworkcontroller/Dockerfile b/build/frameworkcontroller/Dockerfile
@@ -20,15 +20,24 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE
 
-FROM golang:alpine
+FROM golang:alpine as builder
 
 ENV PROJECT_DIR=${GOPATH}/src/github.com/microsoft/frameworkcontroller
+ENV INSTALL_DIR=/opt/frameworkcontroller/frameworkcontroller
 
-RUN apk update && apk add bash && mkdir -p ${PROJECT_DIR}
+RUN apk update && apk add --no-cache bash && \
+  mkdir -p ${PROJECT_DIR} ${INSTALL_DIR}
 COPY . ${PROJECT_DIR}
-WORKDIR ${PROJECT_DIR}
+RUN ${PROJECT_DIR}/build/frameworkcontroller/go-build.sh && \
+  mv ${PROJECT_DIR}/dist/frameworkcontroller/* ${INSTALL_DIR}
 
-RUN ./build/frameworkcontroller/go-build.sh
-WORKDIR ./dist/frameworkcontroller
+
+FROM alpine:latest
+
+ENV INSTALL_DIR=/opt/frameworkcontroller/frameworkcontroller
+
+RUN apk update && apk add --no-cache bash
+COPY --from=builder ${INSTALL_DIR} ${INSTALL_DIR}
+WORKDIR ${INSTALL_DIR}
 
 ENTRYPOINT ["./start.sh"]
diff --git a/build/frameworkcontroller/go-build.sh b/build/frameworkcontroller/go-build.sh
@@ -39,7 +39,7 @@ mkdir -p ${DIST_DIR}
 go build -o ${DIST_DIR}/frameworkcontroller cmd/frameworkcontroller/*
 chmod a+x ${DIST_DIR}/frameworkcontroller
 cp -r bin/frameworkcontroller/* ${DIST_DIR}
-cp -r example/config/default/* ${DIST_DIR}
+cp -r example/config/default/frameworkcontroller.yaml ${DIST_DIR}
 
 echo Succeeded to build binary distribution into ${DIST_DIR}:
 cd ${DIST_DIR} && ls -lR .
diff --git a/cmd/frameworkbarrier/main.go b/cmd/frameworkbarrier/main.go
@@ -0,0 +1,36 @@
+// MIT License
+//
+// Copyright (c) Microsoft Corporation. All rights reserved.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE
+
+package main
+
+import (
+	"github.com/microsoft/frameworkcontroller/pkg/common"
+	"github.com/microsoft/frameworkcontroller/pkg/barrier"
+)
+
+func init() {
+	common.InitAll()
+}
+
+func main() {
+	barrier.NewFrameworkBarrier().Run()
+}
diff --git a/cmd/frameworkcontroller/main.go b/cmd/frameworkcontroller/main.go
@@ -38,7 +38,7 @@ func main() {
 	stopCh := make(chan struct{})
 	defer close(stopCh)
 
-	go controller.NewQueueFrameworkController().Run(stopCh)
+	go controller.NewFrameworkController().Run(stopCh)
 
 	sigTerm := make(chan os.Signal, 1)
 	signal.Notify(sigTerm, syscall.SIGTERM)

diff --git a/doc/known-issue-and-upcoming-feature.md b/doc/known-issue-and-upcoming-feature.md
@@ -13,7 +13,6 @@
    Tracked in [Dashboard errors if pod's owner reference is not supported](https://github.com/kubernetes/dashboard/issues/3251)
 
 ## <a name="UpcomingFeature">Upcoming Feature</a>
-- [ ] Add Distributed TensorFlow Training Example
 - [ ] Support Framework Spec Update
 - [ ] Support Framework Spec Validation and Defaulting
 - [ ] Support Framework Status Subresource

diff --git a/doc/user-manual.md b/doc/user-manual.md
@@ -6,6 +6,7 @@
    - [CompletionCode Convention](#CompletionCodeConvention)
    - [RetryPolicy](#RetryPolicy)
    - [FrameworkAttemptCompletionPolicy](#FrameworkAttemptCompletionPolicy)
+   - [Controller Extension](#ControllerExtension)
    - [Best Practice](#BestPractice)
 
 ## <a name="FrameworkInterop">Framework Interop</a>
@@ -146,7 +147,7 @@ Watch the change events of all Frameworks (in the specified FrameworkNamespace).
 [Container EnvironmentVariable](../pkg/apis/frameworkcontroller/v1/constants.go)
 
 ## <a name="CompletionCodeConvention">CompletionCode Convention</a>
-[CompletionCode Convention](../pkg/apis/frameworkcontroller/v1/types.go)
+[CompletionCode Convention](../pkg/apis/frameworkcontroller/v1/constants.go)
 
 ## <a name="RetryPolicy">RetryPolicy</a>
 ### <a name="RetryPolicy_Spec">Spec</a>
@@ -301,5 +302,11 @@ Notes:
   </tbody>
 </table>
 
+## <a name="ControllerExtension">Controller Extension</a>
+### <a name="FrameworkBarrier">FrameworkBarrier</a>
+1. [Usage](../pkg/barrier/barrier.go)
+2. [Build](../build/frameworkbarrier)
+3. Example: [FrameworkBarrier Example](../example/framework/extension/frameworkbarrier.yaml), [Tensorflow Example](../example/framework/scenario/tensorflow), [etc](../example/framework/scenario).
+
 ## <a name="BestPractice">Best Practice</a>
 [Best Practice](../pkg/apis/frameworkcontroller/v1/types.go)
diff --git a/example/framework/basic/batchfailedpermanent.yaml b/example/framework/basic/batchfailedpermanent.yaml
@@ -10,25 +10,25 @@ spec:
     fancyRetryPolicy: true
     maxRetryCount: 1
   taskRoles:
-    - name: worker
-      taskNumber: 1
-      frameworkAttemptCompletionPolicy:
-        minFailedTaskCount: 1
-        minSucceededTaskCount: -1
-      task:
-        retryPolicy:
-          fancyRetryPolicy: true
-          maxRetryCount: 1
-        pod:
-          spec:
-            restartPolicy: Never
-            containers:
-              - name: ubuntu
-                image: ubuntu:trusty
-                # See CompletionCode Convention in
-                # ./pkg/apis/frameworkcontroller/v1/constants.go
-                command: [
-                "sh", "-c",
-                "sleep 10 &&
-                echo exit with permanent failure to tell controller not to retry &&
-                exit 210"]
+  - name: worker
+    taskNumber: 1
+    frameworkAttemptCompletionPolicy:
+      minFailedTaskCount: 1
+      minSucceededTaskCount: -1
+    task:
+      retryPolicy:
+        fancyRetryPolicy: true
+        maxRetryCount: 1
+      pod:
+        spec:
+          restartPolicy: Never
+          containers:
+          - name: ubuntu
+            image: ubuntu:trusty
+            # See CompletionCode Convention in
+            # ./pkg/apis/frameworkcontroller/v1/constants.go
+            command: [
+            "sh", "-c",
+            "sleep 10 &&
+            echo exit with permanent failure to tell controller not to retry &&
+            exit 210"]