Poetry: missing .serverless/requirements.txt

[revmischa]

Using poetry, normal pyproject.toml file and poetry.lock. Version 5.1.0.
sls package gives me:

Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
 
  Error --------------------------------------------------
 
  Error: ENOENT: no such file or directory, open '/Users/cyber/dev/jb/project/.serverless/requirements.txt'
      at Object.openSync (fs.js:439:3)
      at Object.readFileSync (fs.js:344:35)
      at getRequirements (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:354:6)
      at filterRequirementsFile (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:381:24)
      at generateRequirementsFile (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:67:5)
      at installRequirementsIfNeeded (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:501:3)
      at ServerlessPythonRequirements.installAllRequirements (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:635:29)
  From previous event:
      at PluginManager.invoke (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at getHooks.reduce.then (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/classes/PluginManager.js:525:24)
  From previous event:
      at PluginManager.run (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/classes/PluginManager.js:525:8)
      at variables.populateService.then (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/Serverless.js:133:33)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:120:23)
  From previous event:
      at Serverless.run (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/Serverless.js:120:74)
      at serverless.init.then (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/bin/serverless.js:82:30)
      at /Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:136:16
      at /Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:57:14
      at FSReqWrap.oncomplete (fs.js:141:20)
  From previous event:
      at initializeErrorReporter.then (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/bin/serverless.js:82:8)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:120:23)
  From previous event:
      at Object.<anonymous> (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/bin/serverless.js:71:4)
      at Module._compile (internal/modules/cjs/loader.js:689:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
      at Module.load (internal/modules/cjs/loader.js:599:32)
      at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
      at Function.Module._load (internal/modules/cjs/loader.js:530:3)
      at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
      at startup (internal/bootstrap/node.js:282:19)
      at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              10.14.2
     Framework Version:         1.66.0
     Plugin Version:            3.5.0
     SDK Version:               2.3.0
     Components Version:        2.22.3

[BwL1289]

I am receiving this error for sls deploy and not just for poetry

[AndrewFarley]

@BwL1289 Could you post your serverless.yaml somewhere, your requirements.txt, along with info on your OS and version of NodeJS and version of this plugin?

@revmischa Could you also post your serverless.yaml, and requirements.txt, along with your version of this plugin?

Make sure you're using the latest version of this plugin, and post a little more info so we can replicate it and debug/fix this.

[BwL1289]

Thanks @AndrewFarley!
serverless.yml (i have also slimmed this down to the minimum and still receive the same error)

---
service: space-debris-gui

frameworkVersion: ">=1.34 <2.0.0"

plugins:
  - serverless-python-requirements
  - serverless-plugin-split-stacks
  - serverless-offline
custom:
  pythonRequirements:
    pythonBin: ../../.pyenv/versions/3.7.2/bin/python3.7 
    dockerizePip: true
    zip: true
    slim: true
    slimPatterns:
      - "**/*.egg-info*"
    noDeploy: 
      - pytest
      - tox
  splitStacks:
    nestedStackCount: 20
    perFunction: false
    perType: false
    perGroupFunction: true

provider:
  name: aws
  runtime: python3.7
  stage: dev
  region: us-east-1

  environment:
    MONGO_DB_NAME: ${env:MONGO_DB_NAME}
    MONGO_DB_URL: ${env:MONGO_DB_URL}
  iamManagedPolicies:
    - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"

package:
  individually: true
  exclude:
    - node_modules/**
    - venv/**
    - package.json
    - package-lock.json
    - requirements.txt

functions:
  - ${file(app/routes/aircrafts/functions.yml)}

requirements.txt

apscheduler
black
boto3
botocore
Click==7.0
dnspython==1.16.0
docker
dynaconf[all]
flake8
h5py
matplotlib
mypy
pandas
pre-commit
pymongo==3.9.0
requests
setup-cfg-fmt
setuptools
sphinx
termcolor
towncrier
tqdm
wheel

sls deploy

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command dev
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
 
  Error --------------------------------------------------
 
  Error: ENOENT: no such file or directory, open '/Users/benjaminleff/.vscode/space-debris/.serverless/requirements.txt'
      at Object.openSync (fs.js:440:3)
      at Object.readFileSync (fs.js:342:35)
      at getRequirements (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:354:6)
      at filterRequirementsFile (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:381:24)
      at generateRequirementsFile (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:67:5)
      at installRequirementsIfNeeded (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:501:3)
      at ServerlessPythonRequirements.installAllRequirements (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:635:29)
  From previous event:
      at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:510:17)
      at Deploy.<anonymous> (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:115:50)
  From previous event:
      at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:100:30)
      at /usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:55
  From previous event:
      at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at /usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:525:24
  From previous event:
      at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:525:8)
      at /usr/local/lib/node_modules/serverless/lib/Serverless.js:133:33
      at processImmediate (internal/timers.js:439:21)
      at process.topLevelDomainCallback (domain.js:131:23)
  From previous event:
      at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:120:74)
      at /usr/local/lib/node_modules/serverless/bin/serverless.js:82:30
      at /usr/local/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:136:16
      at /usr/local/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:57:14
      at FSReqCallback.oncomplete (fs.js:146:23)
  From previous event:
      at /usr/local/lib/node_modules/serverless/bin/serverless.js:82:8
      at processImmediate (internal/timers.js:439:21)
      at process.topLevelDomainCallback (domain.js:131:23)
  From previous event:
      at Object.<anonymous> (/usr/local/lib/node_modules/serverless/bin/serverless.js:71:4)
      at Module._compile (internal/modules/cjs/loader.js:956:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
      at Module.load (internal/modules/cjs/loader.js:812:32)
      at Function.Module._load (internal/modules/cjs/loader.js:724:14)
      at Function.Module.runMain (internal/modules/cjs/loader.js:1025:10)
      at internal/main/run_main_module.js:17:11
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              12.13.0
     Framework Version:         1.67.0
     Plugin Version:            3.6.1
     SDK Version:               2.3.0
     Components Version:        2.22.3

[BwL1289]

@AndrewFarley For reference, here is the stripped down version of serverless.yml

---
service: space-debris-gui

frameworkVersion: ">=1.34 <2.0.0"

plugins:
  - serverless-python-requirements

custom:
#   stage: ${opt:stage, self:provider.stage}
  pythonRequirements:
    dockerizePip: true # enables plugin to install package in docker  ${self:custom.isCI.${env:CI}, self:custom.isCI.false}

provider:
  name: aws
  runtime: python3.7


functions:
  - ${file(app/routes/aircrafts/functions.yml)}

[AndrewFarley]

@BwL1289 i tried replicating your problem but wasn't able to. What version of this plugin are you using? I think you're using an old version.

Here's my log running what you have here above...

# serverless package
Serverless: Generated requirements from /root/farley/slstest/requirements.txt in /root/farley/slstest/.serverless/requirements.txt...
Serverless: Installing requirements from /root/.cache/serverless-python-requirements/a630932a3ab4fa1bd748aa36ca92d5b597c8d78f2f6d888cc2bebc1de8a43f89_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.7
Serverless: Using download cache directory /root/.cache/serverless-python-requirements/downloadCacheslspyc
Serverless: Running docker run --rm -v /root/.cache/serverless-python-requirements/a630932a3ab4fa1bd748aa36ca92d5b597c8d78f2f6d888cc2bebc1de8a43f89_slspyc\:/var/task\:z -v /root/.cache/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z lambci/lambda\:build-python3.7 /bin/sh -c 'chown -R 0\\:0 /var/useDownloadCache && python3.7 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 0\\:0 /var/task && chown -R 0\\:0 /var/useDownloadCache'...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
# 

with my package.json...

{
  "name": "test",
  "version": "1.0.0",
  "description": "TEST",
  "author": "Farley <farley@invalid-domain-goes-here.com>",
  "license": "Private",
  "dependencies": {
    "serverless-python-requirements": "github:unitedincome/serverless-python-requirements#master"
  }
}

Confirmed this works on my Mac, and on Linux (ubuntu 18)

[BwL1289]

@AndrewFarley appreciate the help. I'm using 5.1.0.

Here is my package.json

{
  "name": "space-debris-gui",
  "description": "",
  "version": "0.1.0",
  "dependencies": {},
  "devDependencies": {
    "serverless": "^1.67.0",
    "serverless-offline": "^6.1.2",
    "serverless-plugin-split-stacks": "^1.9.3",
    "serverless-python-requirements": "^5.1.0"
  }
}

I also tried to changing it to reflect what you have: "serverless-python-requirements": "github:unitedincome/serverless-python-requirements#master" but am receiving the same error.

If it helps, here's the relevant info from package-lock.json

"serverless-python-requirements": {
      "version": "5.1.0",
      "resolved": "https://registry.npmjs.org/serverless-python-requirements/-/serverless-python-requirements-5.1.0.tgz",
      "integrity": "sha512-lJhikc6wJsOYFxYNGK763xtQITrtGPHD2KJvs8qaBkkOET4RVk8E0N3d8wAO22k+Tk7N6a6Jonnvsmo3W8vtCA==",
      "dev": true,
      "requires": {
        "@iarna/toml": "^2.2.3",
        "appdirectory": "^0.1.0",
        "bluebird": "^3.0.6",
        "fs-extra": "^7.0.0",
        "glob-all": "^3.1.0",
        "is-wsl": "^2.0.0",
        "jszip": "^3.1.0",
        "lodash.get": "^4.4.2",
        "lodash.set": "^4.3.2",
        "lodash.uniqby": "^4.0.0",
        "lodash.values": "^4.3.0",
        "rimraf": "^3.0.2",
        "sha256-file": "1.0.0",
        "shell-quote": "^1.6.1"
      }
    },

What other information can I provide that may help to debug this? Again, really appreciate the help as this is a blocker.

[AndrewFarley]

@BwL1289 I tried 5.1.0 and same, no issue here. Could you rm -Rf node_modules && rm -f package-lock.json && npm install then run serverless package and copy/paste me your output from that last command. (with the modified package.json to use the #master as I had above)

[BwL1289]

@AndrewFarley update - i had a pyproject.toml file in my root dir that i just removed (it is being used for black configs, towncrier configs, and setuptools configs and it seems to be running as expected (still in progress).

I assume that the plugin doesnt support both files simulteanously (perhaps without additional configuration). is there a way to make them work together?

[AndrewFarley]

@BwL1289 If you have a pyproject.toml file it uses poetry instead of pip. But definitely copy/paste me your log from serverless package

[BwL1289]

@AndrewFarley thanks so much!

$ serverless package

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command dev
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /Users/benjaminleff/.vscode/space-debris/requirements.txt in /Users/benjaminleff/.vscode/space-debris/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/benjaminleff/Library/Caches/serverless-python-requirements/a630932a3ab4fa1bd748aa36ca92d5b597c8d78f2f6d888cc2bebc1de8a43f89_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.7
Serverless: Using download cache directory /Users/benjaminleff/Library/Caches/serverless-python-requirements/downloadCacheslspyc
Serverless: Running docker run --rm -v /Users/benjaminleff/Library/Caches/serverless-python-requirements/a630932a3ab4fa1bd748aa36ca92d5b597c8d78f2f6d888cc2bebc1de8a43f89_slspyc\:/var/task\:z -v /Users/benjaminleff/Library/Caches/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z -u 0 lambci/lambda\:build-python3.7 python3.7 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage

[AndrewFarley]

@BwL1289 Looks good eh?

[BwL1289]

@AndrewFarley yes sir! thanks again for the help. Appreciate you guys and all that you do. Will try to find workarounds for the configs specified in pyproject.toml

[AndrewFarley]

Yeah @BwL1289 I don't have much experience with poetry, but it sounds like both of you had the same problem, so it appears there is probably a valid bug here somewhere. Hopefully someone with poetry experience can follow up on this.