Cannot use Deployer with submodules

[FelipeEmerim]

Before submitting your issue, please make sure that you've checked all of the checkboxes below.

• You're running the latest release version of Deployer.
• Ensure that you're running at least PHP 7.0.8, you can check this by running php -v
• You've ran composer install --no-dev from the root of your installation.
• You've ran npm install --production from the root of your installation.
• You've ran rm -rf bootstrap/cache/* from the root of your installation.
• You have restarted the queue listener and node socket server.

Describe the bug

When trying to use deployer with a repository that has submodules. The deploy fails with the following error: permission denied (publickey) while cloning the submodule.

Expected behaviour

It would properly deploy the project.

Actual behaviour

The deploy fails due to the error mentioned above without event entering the first step.

Steps to reproduce

• Include a project with submodule in deployer.
• Configure the same ssh key in the master and submodule projects.
• Try to deploy a version of the project.

Environment info

Laravel Environment

• Laravel Version: 5.5.41
• Timezone: America/Sao_Paulo
• Debug Mode: ✔
• Storage Dir Writable: ✔
• Cache Dir Writable: ✔
• Decomposer Version: ^1.0
• App Size: 2 GB

Server Environment

• PHP Version: 7.0.33-0+deb9u3
• Server Software: nginx/1.10.3
• Server OS: Linux magneto 3.16.0-6-amd64 Route models #1 SMP Debian 3.16.57-2 (2018-07-14) x86_64
• Database: pgsql
• SSL Installed: ✘
• Cache Driver: file
• Session Driver: file
• Openssl Ext: ✔
• PDO Ext: ✔
• Mbstring Ext: ✔
• Tokenizer Ext: ✔
• XML Ext: ✘
• Curl Ext: ✘
• GD Ext: ✔
• JSON Ext: ✔

Installed Packages & their version numbers

• andywer/js-localization : dev-laravel-5
• backup-manager/laravel : ^1.1
• creativeorange/gravatar : ~1.0
• doctrine/dbal : 2.5.13
• fideloper/proxy : ^3.3
• graham-campbell/binput : ~4.0
• guzzlehttp/guzzle : ^6.2
• htmlmin/htmlmin : ~5.0
• intervention/image : ^2.3
• laracademy/interactive-make : ^1.1
• laravel-notification-channels/hipchat : ~1.0
• laravel-notification-channels/twilio : ~2.0.1
• laravel-notification-channels/webhook : ~1.1.0
• laravel/framework : ^5.5
• laravel/tinker : 1.0.*
• lubusin/laravel-decomposer : ^1.0
• mccool/laravel-auto-presenter : ^6.2
• melihovv/laravel-log-viewer : ^5.5
• michele-angioni/multi-language : 0.4
• pda/pheanstalk : ~3.1
• pragmarx/google2fa : ^1.0.0
• predis/predis : ~1.0
• rebelinblue/laravel5-zxcvbn : ^1.0.0
• tymon/jwt-auth : ~0.5
• version/version : ^2.2

Extra Information

• proc_open enabled : ✔

Logs (see storage/logs/) or other output that would be helpful

[2019-09-05 20:26:38] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Mirror git repository
chmod +x /var/www/deployer/storage/app/tmp/ssh5W8wC8
export GIT_SSH=/var/www/deployer/storage/app/tmp/ssh5W8wC8

if [ ! -d /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git ]; then
    git clone --mirror git@bitbucket.org:VE3/colplan.git /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git
fi

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git fetch --all --prune
EOF
  
[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Get commit details - 1365
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git log master -n1 --pretty=format:"%H%x09%an%x09%ae"
EOF
  
[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Create release archive - 1365
git clone --depth 1 --recursive /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git /var/www/deployer/storage/app/tmp/clone_17_20190905202638
cd /var/www/deployer/storage/app/tmp/clone_17_20190905202638
git checkout 36cc87b2cbb01f96473c80abce76c85081a871f0
/var/www/deployer/resources/scripts/tools/GitArchiveAll.sh --tree-ish 36cc87b2cbb01f96473c80abce76c85081a871f0 --format tar.gz --verbose /var/www/deployer/storage/app/17_20190905202636.tar.gz
cd -
rm -rf /var/www/deployer/storage/app/tmp/clone_17_20190905202638
EOF
  
[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### List git references
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git tag --list --no-column
EOF
  
[2019-09-05 20:26:43] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### List git references
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git branch --list --no-column
EOF
  
[2019-09-05 20:26:45] development.ERROR: Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638'...
warning: --depth is ignored in local clones; use file:// instead.
done.
Submodule 'forecaster' (git@bitbucket.org:VE3/forecaster.git) registered for path 'forecaster'
Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@bitbucket.org:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed
Failed to clone 'forecaster'. Retry scheduled
Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@bitbucket.org:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed
Failed to clone 'forecaster' a second time, aborting

When nailing down the issue I found that cloning a submodule from a mirror requires authentication. So the ssh configs script should be passed as an argument to the CreateReleaseArchive.sh which should export GIT_SSH as you do when you clone the mirror.

If you do not need a submodule just remove --recursive from the git clone command.

[REBELinBLUE]

Have you made sure the deploy key is added to the sub module repository?

…

----------------- Stephen Ball stephen@rebelinblue.com

On 6 Sep 2019, 21:19 +0100, FelipeEmerim ***@***.***>, wrote: Before submitting your issue, please make sure that you've checked all of the checkboxes below. • [ X ] You're running the latest release version of Deployer. • [ X ] Ensure that you're running at least PHP 7.0.8, you can check this by running php -v • [ X ] You've ran composer install --no-dev from the root of your installation. • [ X ] You've ran npm install --production from the root of your installation. • [ X ] You've ran rm -rf bootstrap/cache/* from the root of your installation. • [ X ] You have restarted the queue listener and node socket server. Describe the bug When trying to use deployer with a repository that has submodules. The deploy fails with the following error: permission denied (publickey) while cloning the submodule. Expected behaviour It would properly deploy the project. Actual behaviour The deploy fails due to the error mentioned above without event entering the first step. Steps to reproduce • Include a project with submodule in deployer. • Configure the same ssh key in the master and submodule projects. • Try to deploy a version of the project. Environment info Laravel Environment • Laravel Version: 5.5.41 • Timezone: America/Sao_Paulo • Debug Mode: ✔ • Storage Dir Writable: ✔ • Cache Dir Writable: ✔ • Decomposer Version: ^1.0 • App Size: 2 GB Server Environment • PHP Version: 7.0.33-0+deb9u3 • Server Software: nginx/1.10.3 • Server OS: Linux magneto 3.16.0-6-amd64 #1 SMP Debian 3.16.57-2 (2018-07-14) x86_64 • Database: pgsql • SSL Installed: ✘ • Cache Driver: file • Session Driver: file • Openssl Ext: ✔ • PDO Ext: ✔ • Mbstring Ext: ✔ • Tokenizer Ext: ✔ • XML Ext: ✘ • Curl Ext: ✘ • GD Ext: ✔ • JSON Ext: ✔ Installed Packages & their version numbers • andywer/js-localization : dev-laravel-5 • backup-manager/laravel : ^1.1 • creativeorange/gravatar : ~1.0 • doctrine/dbal : 2.5.13 • fideloper/proxy : ^3.3 • graham-campbell/binput : ~4.0 • guzzlehttp/guzzle : ^6.2 • htmlmin/htmlmin : ~5.0 • intervention/image : ^2.3 • laracademy/interactive-make : ^1.1 • laravel-notification-channels/hipchat : ~1.0 • laravel-notification-channels/twilio : ~2.0.1 • laravel-notification-channels/webhook : ~1.1.0 • laravel/framework : ^5.5 • laravel/tinker : 1.0.* • lubusin/laravel-decomposer : ^1.0 • mccool/laravel-auto-presenter : ^6.2 • melihovv/laravel-log-viewer : ^5.5 • michele-angioni/multi-language : 0.4 • pda/pheanstalk : ~3.1 • pragmarx/google2fa : ^1.0.0 • predis/predis : ~1.0 • rebelinblue/laravel5-zxcvbn : ^1.0.0 • tymon/jwt-auth : ~0.5 • version/version : ^2.2 Extra Information • proc_open enabled : ✔ Logs (see storage/logs/) or other output that would be helpful [2019-09-05 20:26:38] development.DEBUG: ### Run script locally bash -s << 'EOF' # Turn on quit on non-zero exit set -e ### Mirror git repository chmod +x /var/www/deployer/storage/app/tmp/ssh5W8wC8 export GIT_SSH=/var/www/deployer/storage/app/tmp/ssh5W8wC8 if [ ! -d /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git ]; then git clone --mirror ***@***.***:VE3/colplan.git /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git fi cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git git fetch --all --prune EOF [2019-09-05 20:26:41] development.DEBUG: ### Run script locally bash -s << 'EOF' # Turn on quit on non-zero exit set -e ### Get commit details - 1365 cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git git log master -n1 --pretty=format:"%H%x09%an%x09%ae" EOF [2019-09-05 20:26:41] development.DEBUG: ### Run script locally bash -s << 'EOF' # Turn on quit on non-zero exit set -e ### Create release archive - 1365 git clone --depth 1 --recursive /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git /var/www/deployer/storage/app/tmp/clone_17_20190905202638 cd /var/www/deployer/storage/app/tmp/clone_17_20190905202638 git checkout 36cc87b2cbb01f96473c80abce76c85081a871f0 /var/www/deployer/resources/scripts/tools/GitArchiveAll.sh --tree-ish 36cc87b2cbb01f96473c80abce76c85081a871f0 --format tar.gz --verbose /var/www/deployer/storage/app/17_20190905202636.tar.gz cd - rm -rf /var/www/deployer/storage/app/tmp/clone_17_20190905202638 EOF [2019-09-05 20:26:41] development.DEBUG: ### Run script locally bash -s << 'EOF' # Turn on quit on non-zero exit set -e ### List git references cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git git tag --list --no-column EOF [2019-09-05 20:26:43] development.DEBUG: ### Run script locally bash -s << 'EOF' # Turn on quit on non-zero exit set -e ### List git references cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git git branch --list --no-column EOF [2019-09-05 20:26:45] development.ERROR: Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638'... warning: --depth is ignored in local clones; use file:// instead. done. Submodule 'forecaster' ***@***.***:VE3/forecaster.git) registered for path 'forecaster' Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'... Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. fatal: clone of ***@***.***:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed Failed to clone 'forecaster'. Retry scheduled Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'... Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. fatal: clone of ***@***.***:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed Failed to clone 'forecaster' a second time, aborting When nailing down the issue I found that cloning a submodule from a mirror requires authentication. So the ssh configs script should be passed as an argument to the CreateReleaseArchive.sh which should export GIT_SSH as you do when you clone the mirror. If you do not need a submodule just remove --recursive from the git clone command. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[FelipeEmerim]

Yes, same private key on the two repos. I can clone them as long as I pass the ssh key. It seems that the deployer does not use the wrapper ssh script when cloning from the mirror which raises the error.

[FelipeEmerim]

A simple test:

#!/usr/bin/env sh

ssh -o CheckHostIP=no \
    -o IdentitiesOnly=yes \
    -o StrictHostKeyChecking=no \
    -o PasswordAuthentication=no \
    -o IdentityFile=MyprivateKey $*`

export GIT_SSH=/path/to/script.sh

git clone --mirror git@bitbucket.org:VE3/repoWithSubmodules.git

This is what deployer does to clone the mirror and it is working. Now when it clones from the mirror to the repository to archive, it does not set the git_ssh var so:

unset GIT_SSH

git clone --depth 1 --recursive projectWithSubmdules.git projectWithSubmodules

This last command will raise the mentioned error.

[vhlavsa]

Hi everybody, I have a same problem. Project with submodules, all of them have access keys same as the main project, but deploy keeps failing.

Is there any way to make it work without getting rid of the submodules?

Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.
fatal: clone of 'git@bitbucket.org:xyz.git' into submodule path 'xyz' failed

[FelipeEmerim]

I tried some workarounds but none of them worked. In our case, since the submodule is actually a SOAP service we deployed it separately.

[darmen]

Hey there 👋🏼

I've encountered similar problem some time ago, and ended up writing a quick tip on how I approached it – https://heydarmen.com/blog/using-deployer-with-git-submodules.html

[REBELinBLUE]

hey @darmen I think you are looking for https://github.com/deployphp/deployer

[darmen]

Sorry @REBELinBLUE.

[REBELinBLUE]

No worries :)