[tfjs-converter] Bazel build migration (#5388)

FEATURE
INTERNAL
* init check in for converter bazel migration

* fixed build failure

* fix test target error

* tests are passing

* added test_snippets target

* add gen_json rule

* create gen-json bazel rule

* added build rules for converter pyton package

* added python runtimes and update pip test script

* fixed dupe deps

* updated cloud build

* fixed bazel lint error

* fix test generated cloudbuild

* use bazel installed python interpreter for pip_install;

* fix bazel format

* fix pip test

* update converter npm link

* add link package for tfjs-converter

* Generate converter files needed for the build with Bazel (#5393)


* Generate .ts op files from json

* Generate kernel2op with bazel

* Remove console.logs

* Bazel lint

* Build all ops with a single target

* Update contributing missing op readme

* Bazel lint

* fixed converter python tests

* fix failed test

* Update converter entrypoints (#5395)


* Update converter entrypoints

* Add converter to 'reinstall' in link-package

* fixed tfjs union package test

* remove gen_json rule

* fix bazel lint

* fix lint

* Add kernel2ops to converter pkg (#5399)

* Add kernel2ops to converter pkg

* remove copy metadata target

* disable python toolchain registration

* added py toolchain back

* update th link package

* remove numpy requirement

* disable python runtime registration

* register the python runtimes

* update the make-version script to update the python BUILD file for the package.json version

* updated converter nightly test cloud build

* fixed the cloud build file

* build the wheel in the test script

* use root dir

* fixed tests

* fixes

* update iamge

* fix yml

* revert

* fixed cloudbuild test

* list the pip files

* update cloudbuild

* cp instead of mv

* adding more log

* update log

* update log

* update log

* Use a different symlink prefix for wasm builds (#5455)

FEATURE

* removed debug logging

* limit numpy version

* increase nightly time limit

* added converter to bazel packages

* address the comments

* fixed js and python tests

* addressed comments

* fixed build file test

* fix bazel format

* fix typo

* fix tfjs tool test

Co-authored-by: Matthew Soulanille <[email protected]>

Author: pyu10055

CONTRIBUTING_MISSING_OP.md

@@ -71,18 +71,14 @@ In `run_tests.ts`, `IGNORE_LIST`, add test_name to the list.
 * [WASM backend Round](https://github.com/tensorflow/tfjs/pull/4486)
 
 ### Step 3. Add the op to Converter’s executor.
-1. Add op mapping in the op list ts file, use your best judgement to assign an op category: `tfjs-converter/src/operations/op_list/{corresponding_op_category}.ts`. Use
+1. Add op mapping in the op list ts file, use your best judgement to assign an op category: `tfjs-converter/python/tensorflowjs/op_list/{corresponding_op_category}.json`. The corresponding `.ts` file will be automatically generated when converter is built. Use
 the [TF C++ op api](https://www.tensorflow.org/api_docs/cc/) as reference for tfOpName, inputs, and attrs.
 
-2. Auto generates corresponding json file by running following command in directory tfjs-converter: `yarn gen-json`. Check that the json object is generated in `python/tensorflowjs/op_list/{corresponding_op_group}.json`
+2. Find the corresponding executor for the op and add the op to the switch, the [executors](https://github.com/tensorflow/tfjs/tree/master/tfjs-converter/src/operations/executors) are in `tfjs-converter/src/operations/executors`.
 
-3. Find the corresponding executor for the op and add the op to the switch, the [executors](https://github.com/tensorflow/tfjs/tree/master/tfjs-converter/src/operations/executors) are in `tfjs-converter/src/operations/executors`.
+3. Add a test to the corresponding executor test file.
 
-4. Add a test to the corresponding executor test file.
-
-5. Update the supported op doc in `tfjs-converter/docs/supported_ops.md`.
-
-6. Add a mapping in kernel2op.json.
+4. Update the supported op doc in `tfjs-converter/docs/supported_ops.md`.
 
 **Example PRs**
 * [SparseReshape Op](https://github.com/tensorflow/tfjs/pull/4963)

WORKSPACE

@@ -209,3 +209,89 @@ http_archive(
 load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
 
 bazel_skylib_workspace()
+
+# Special logic for building python interpreter with OpenSSL from homebrew.
+# See https://devguide.python.org/setup/#macos-and-os-x
+_py3_configure = """
+if [[ "$OSTYPE" == "darwin"* ]]; then
+    ./configure --prefix=$(pwd)/bazel_install_py3 --with-openssl=$(brew --prefix openssl)
+else
+    ./configure --prefix=$(pwd)/bazel_install_py3
+fi
+"""
+
+_py2_configure = """
+if [[ "$OSTYPE" == "darwin"* ]]; then
+    ./configure --prefix=$(pwd)/bazel_install_py2 --with-openssl=$(brew --prefix openssl)
+else
+    ./configure --prefix=$(pwd)/bazel_install_py2
+fi
+"""
+
+http_archive(
+    name = "python3_interpreter",
+    build_file_content = """
+exports_files(["python3_bin"])
+filegroup(
+    name = "files",
+    srcs = glob(["bazel_install_py3/**"], exclude = ["**/* *"]),
+    visibility = ["//visibility:public"],
+)
+""",
+    patch_cmds = [
+        "mkdir $(pwd)/bazel_install_py3",
+        _py3_configure,
+        "make",
+        "make install",
+        "ln -s bazel_install_py3/bin/python3 python3_bin",
+    ],
+    sha256 = "fb1a1114ebfe9e97199603c6083e20b236a0e007a2c51f29283ffb50c1420fb2",
+    strip_prefix = "Python-3.8.11",
+    urls = ["https://www.python.org/ftp/python/3.8.11/Python-3.8.11.tar.xz"],
+)
+
+http_archive(
+    name = "python2_interpreter",
+    build_file_content = """
+exports_files(["python_bin"])
+filegroup(
+    name = "files",
+    srcs = glob(["bazel_install_py2/**"], exclude = ["**/* *"]),
+    visibility = ["//visibility:public"],
+)
+""",
+    patch_cmds = [
+        "mkdir $(pwd)/bazel_install_py2",
+        _py2_configure,
+        "make",
+        "make install",
+        "ln -s bazel_install_py2/bin/python python_bin",
+    ],
+    sha256 = "a4f05a0720ce0fd92626f0278b6b433eee9a6173ddf2bced7957dfb599a5ece1",
+    strip_prefix = "Python-2.7.13",
+    urls = ["https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz"],
+)
+
+register_toolchains("//tfjs-converter/python:tfjs_py_toolchain")
+
+http_archive(
+    name = "rules_python",
+    sha256 = "934c9ceb552e84577b0faf1e5a2f0450314985b4d8712b2b70717dc679fdc01b",
+    url = "https://github.com/bazelbuild/rules_python/releases/download/0.3.0/rules_python-0.3.0.tar.gz",
+)
+
+load("@rules_python//python:pip.bzl", "pip_install")
+
+# Create a central external repo, @tensorflowjs_dev_deps, that contains Bazel targets for all the
+# third-party packages specified in the requirements.txt file.
+pip_install(
+    name = "tensorflowjs_dev_deps",
+    python_interpreter_target = "@python3_interpreter//:python3_bin",
+    requirements = "//tfjs-converter/python:requirements-dev.txt",
+)
+
+pip_install(
+    name = "tensorflowjs_deps",
+    python_interpreter_target = "@python3_interpreter//:python3_bin",
+    requirements = "//tfjs-converter/python:requirements.txt",
+)

e2e/benchmarks/movenet/group1-shard1of3.bin

@@ -13,7 +13,7 @@
     "@tensorflow/tfjs-backend-cpu": "link:../link-package/node_modules/@tensorflow/tfjs-backend-cpu",
     "@tensorflow/tfjs-backend-wasm": "link:../tfjs-backend-wasm",
     "@tensorflow/tfjs-backend-webgl": "link:../tfjs-backend-webgl",
-    "@tensorflow/tfjs-converter": "link:../tfjs-converter",
+    "@tensorflow/tfjs-converter": "link:../link-package/node_modules/@tensorflow/tfjs-converter",
     "@tensorflow/tfjs-core": "link:../link-package/node_modules/@tensorflow/tfjs-core",
     "@tensorflow/tfjs-data": "link:../tfjs-data",
     "@tensorflow/tfjs-layers": "link:../tfjs-layers",

e2e/benchmarks/movenet/group1-shard2of3.bin

@@ -54,11 +54,11 @@ fi
 
 # Packages to publish.
 PACKAGES=("tfjs-core" "tfjs-backend-cpu" "tfjs-backend-webgl" \
-"tfjs-backend-wasm" "tfjs-layers" "tfjs-converter" "tfjs-data" "tfjs" \
+"tfjs-backend-wasm" "tfjs-layers" "tfjs-data" "tfjs" \
 "tfjs-node" "tfjs-node-gpu")
 
 # Packages that build with Bazel
-BAZEL_PACKAGES=("tfjs-core" "tfjs-backend-cpu" "tfjs-tflite")
+BAZEL_PACKAGES=("tfjs-core" "tfjs-backend-cpu" "tfjs-tflite" "tfjs-converter")
 
 for package in "${PACKAGES[@]}"
 do

e2e/benchmarks/movenet/group1-shard3of3.bin

@@ -5,14 +5,16 @@
   "license": "Apache 2.0",
   "private": true,
   "scripts": {
-    "build": "yarn build-link-package-core && yarn reinstall",
+    "build": "yarn build-link-package-core && yarn build-converter && yarn reinstall",
     "build-link-package-core": "cd ../link-package-core && yarn build",
+    "build-converter": "cd ../tfjs-converter && yarn && yarn build",
     "reinstall-link-package-core": "cd ../link-package-core && yarn reinstall",
     "reinstall": "yarn && yarn reinstall-link-package-core && rimraf node_modules && yarn"
   },
   "devDependencies": {
     "@tensorflow/tfjs-backend-cpu": "link:../link-package-core/node_modules/@tensorflow/tfjs-backend-cpu",
     "@tensorflow/tfjs-core": "link:../link-package-core/node_modules/@tensorflow/tfjs-core",
+    "@tensorflow/tfjs-converter": "file:../dist/bin/tfjs-converter/tfjs-converter_pkg",
     "rimraf": "^3.0.2"
   }
 }

e2e/benchmarks/movenet/model.json

@@ -6,6 +6,9 @@
   version "0.0.0"
   uid ""
 
+"@tensorflow/tfjs-converter@file:../dist/bin/tfjs-converter/tfjs-converter_pkg":
+  version "0.0.0"
+
 "@tensorflow/tfjs-core@link:../link-package-core/node_modules/@tensorflow/tfjs-core":
   version "0.0.0"
   uid ""

e2e/package.json

@@ -19,11 +19,14 @@
     "@types/node-fetch": "~2.1.2",
     "@types/seedrandom": "^2.4.28",
     "@types/shelljs": "^0.8.7",
+    "ajv": "~6.3.0",
     "argparse": "^1.0.10",
     "chalk": "~2.4.2",
     "clang-format": "~1.2.4",
     "console-table-printer": "^2.4.32",
+    "copyfiles": "~1.2.0",
     "core-js": "3",
+    "deep-equal": "^1.0.1",
     "jasmine": "~3.1.0",
     "jasmine-core": "~3.1.0",
     "js-yaml": "^3.14.0",
@@ -36,6 +39,8 @@
     "karma-sourcemap-loader": "^0.3.8",
     "long": "4.0.0",
     "mkdirp": "~0.5.1",
+    "opn": "~5.1.0",
+    "protobufjs": "~6.8.6",
     "requirejs": "^2.3.6",
     "rollup": "^2.46.0",
     "rollup-plugin-sourcemaps": "^0.6.3",
@@ -44,6 +49,7 @@
     "seedrandom": "^3.0.5",
     "shelljs": "~0.8.3",
     "terser": "^5.7.0",
+    "ts-morph": "^11.0.3",
     "ts-node": "~8.8.2",
     "tslint": "~6.1.3",
     "typescript": "3.5.3"

e2e/scripts/publish-tfjs-ci.sh

@@ -63,16 +63,6 @@ steps:
     waitFor:
       - yarn-common
       - yarn-link-package
-  - name: 'node:10'
-    dir: tfjs-converter
-    entrypoint: yarn
-    id: build-tfjs-converter
-    args:
-      - build-ci
-    waitFor:
-      - yarn-tfjs-converter
-      - yarn-common
-      - yarn-link-package
   - name: 'node:10'
     dir: tfjs-converter
     entrypoint: yarn
@@ -84,18 +74,13 @@ steps:
       - yarn-common
       - yarn-link-package
   - name: 'node:10'
-    dir: tfjs-converter
-    entrypoint: yarn
-    id: test-js-tfjs-converter
+    id: 'create-pips-tfjs-converter'
+    entrypoint: 'bash'
     args:
-      - test-ci
+      - './tfjs-converter/scripts/create_python_pips.sh'
     waitFor:
-      - yarn-tfjs-converter
-      - lint-tfjs-converter
       - yarn-common
       - yarn-link-package
-    env:
-      - NIGHTLY=$_NIGHTLY
   - name: gcr.io/google-appengine/python
     dir: tfjs-converter/python
     entrypoint: bash
@@ -105,34 +90,7 @@ steps:
       - '--test'
       - /tmp/tfjs-pips
     waitFor:
-      - yarn-common
-      - yarn-link-package
-  - name: 'python:2'
-    dir: tfjs-converter/python
-    entrypoint: bash
-    id: test-python2-tfjs-converter
-    args:
-      - ./run-python-tests.sh
-    waitFor:
-      - yarn-common
-      - yarn-link-package
-  - name: 'python:3.6'
-    dir: tfjs-converter/python
-    entrypoint: bash
-    id: test-python3.6-tfjs-converter
-    args:
-      - ./run-python-tests.sh
-    waitFor:
-      - yarn-common
-      - yarn-link-package
-  - name: 'node:10'
-    dir: tfjs-converter
-    entrypoint: yarn
-    id: test-snippets-tfjs-converter
-    args:
-      - test-snippets-ci
-    waitFor:
-      - build-tfjs-converter
+      - create-pips-tfjs-converter
       - yarn-common
       - yarn-link-package
   - name: 'node:10'
@@ -412,8 +370,8 @@ steps:
       - yarn-common
       - yarn-link-package
       - yarn-tfjs-converter
-      - build-tfjs-converter
       - lint-tfjs-converter
+      - create-pips-tfjs-converter
       - yarn-tfjs-layers
       - build-tfjs-layers
       - lint-tfjs-layers
@@ -434,8 +392,8 @@ steps:
       - yarn-common
       - yarn-link-package
       - yarn-tfjs-converter
-      - build-tfjs-converter
       - lint-tfjs-converter
+      - create-pips-tfjs-converter
       - yarn-tfjs-layers
       - build-tfjs-layers
       - lint-tfjs-layers
@@ -456,8 +414,8 @@ steps:
       - yarn-common
       - yarn-link-package
       - yarn-tfjs-converter
-      - build-tfjs-converter
       - lint-tfjs-converter
+      - create-pips-tfjs-converter
       - yarn-tfjs-layers
       - build-tfjs-layers
       - lint-tfjs-layers
@@ -479,8 +437,8 @@ steps:
       - yarn-common
       - yarn-link-package
       - yarn-tfjs-converter
-      - build-tfjs-converter
       - lint-tfjs-converter
+      - create-pips-tfjs-converter
       - yarn-tfjs-layers
       - build-tfjs-layers
       - lint-tfjs-layers
@@ -680,8 +638,8 @@ steps:
       - build-tfjs-backend-webgl
       - lint-tfjs-backend-webgl
       - yarn-tfjs-converter
-      - build-tfjs-converter
       - lint-tfjs-converter
+      - create-pips-tfjs-converter
       - yarn-tfjs-data
       - build-tfjs-data
       - lint-tfjs-data
@@ -718,8 +676,8 @@ steps:
       - build-tfjs-backend-webgl
       - lint-tfjs-backend-webgl
       - yarn-tfjs-converter
-      - build-tfjs-converter
       - lint-tfjs-converter
+      - create-pips-tfjs-converter
       - yarn-tfjs-data
       - build-tfjs-data
       - lint-tfjs-data