unable to create-release (can gen manifest, d/l binaries & create blobs fine)

[brybinary]

Hi

following the guide but it fails on creating the release <- I am using bosh v2. binaries/files are where they are supposed to be (afaik).

bosh version: version 2.0.29-1f8ec43-2017-07-14T16:10:31Z
go version: go version go1.9 darwin/amd64
spiff version: spiff version 1.0.7
virtualbox version: 5.1.22r115126
OS: mac osx El Capitan 10.11.6

Error:
user@machine:~/deployments/vbox/splunk-firehose-nozzle-release$ ./scripts/build-release.sh

Building splunk-firehose-nozzle-release

Cleaning up blobs
Downloading binaries
Go package already exist, skipping download
Splunk package already exist, skipping download
Adding blobs
Added blob 'golang'

Succeeded
Added blob 'golang'

Succeeded
Added blob 'splunk'

Succeeded
Added blob 'splunk'

Succeeded
Creating release
Building a release from directory '/Users/USER/deployments/vbox/splunk-firehose-nozzle-release':

• Constructing packages from directory:
  • Reading package from '/Users/USER/deployments/vbox/splunk-firehose-nozzle-release/packages/common':
    Collecting package files:
    Missing files for pattern 'common/**/*'
  • Reading package from '/Users/USER/deployments/vbox/splunk-firehose-nozzle-release/packages/golang':
    Collecting package files:
    Missing files for pattern 'golang/go-linux-amd64.tar.gz'
  • Reading package from '/Users/USER/deployments/vbox/splunk-firehose-nozzle-release/packages/splunk':
    Collecting package files:
    Missing files for pattern 'splunk/splunk-linux-x86_64.tgz'
  • Expected to find package 'golang' since it's a dependency of package 'splunk-nozzle'
• Constructing jobs from manifest:
  • Expected to find package 'splunk' since it's a dependency of job 'splunk-forwarder'
  • Expected to find package 'splunk' since it's a dependency of job 'splunk-full'
  • Expected to find package 'common' since it's a dependency of job 'splunk-nozzle'

Exit code 1

Any pointers appreciated.

Thx

[chenziliang]

could you please go to Splunk-firehose-nozzle repo and follow the instructions in readme to deploy nozzle as a PCF app ?

[brybinary]

Hi chenziliang

I will give that a try and report back.

Thx

[brybinary]

update:

followed deploying the nozzle into cf (https://github.com/cloudfoundry-community/splunk-firehose-nozzle) and it seemed to work ok.

I do still require to deploy splunk via bosh, so will continue looking into why it failed when creating the release. As always, any pointers greatly appreciated!

Here's an output showing the nozzle being deployed into CF (#2), as suggested.

Thx

output
Starting app splunk-firehose-nozzle in org cloudfoundry / space development as admin...
Downloading go_buildpack...
Downloaded go_buildpack (4.6M)
Creating container
Successfully created container
Downloading app package...
Downloaded app package (5.5M)
Staging...
-----> Go Buildpack version 1.8.6
-----> Installing godep v79
-----> Running: go install -tags cloudfoundry -buildmode pie .
Exit status 0
Staging complete
Uploading droplet, build artifacts cache...
Uploading build artifacts cache...
Uploading droplet...
Uploaded build artifacts cache (215B)
Uploaded droplet (8M)
Uploading complete
Stopping instance 7975897a-3cd0-4049-b3a5-bf7075810d88
Destroying container
Successfully destroyed container

2 of 2 instances running

App started
OK
App splunk-firehose-nozzle was started using this command splunk-firehose-nozzle
Showing health and status for app splunk-firehose-nozzle in org cloudfoundry / space development as admin...
OK

requested state: started
instances: 2/2
usage: 512M x 2 instances
urls: splunk-firehose-nozzle-northern-bedspring.bosh-lite.com
last uploaded: Fri Sep 1 17:06:31 UTC 2017
stack: cflinuxfs2
buildpack: go_buildpack

 state     since                    cpu    memory         disk         details

#0 running 2017-09-01 06:07:43 PM 0.0% 564K of 512M 1.3M of 1G
#1 running 2017-09-01 06:07:43 PM 0.0% 0 of 512M 0 of 1G
user@machine:~/deployments/splunk_nozzle$

[brybinary]

update 2:

updated the ./scripts/build-release to the following, which then seemed to allow the release to be created. Just doing some further testing on the deployment. Happy to receive feedback, as to why putting the splunk-linux-x86_64.tgz as a blob for common made this progress through creating release...

my ./scripts/build-release.sh
`#!/usr/bin/env bash

set -e

if [ "$0" != "./scripts/build-release.sh" ]; then
echo "'build-release.sh' should be run from repository root"
exit 1
fi

function usage(){

&2 echo "
Usage:
$0 [version]

Ex:
$0 0+dev.1
"
exit 1
}

if [ "$1" == "-h" ] || [ "$1" == "--help" ] || [ "$1" == "help" ]; then
usage
fi

if [ "$#" -gt 0 ]; then
if [ -e "$1" ]; then
export version=cat $1
else
export version=$1
fi
fi

echo "Building splunk-firehose-nozzle-release ${version}"
echo ""

echo "Cleaning up blobs"
rm -rf .blobs/* blobs/*
echo "--- {} " > config/blobs.yml

echo "Downloading binaries"

mkdir -p tmp

export go_pkg_path=./tmp/go-linux-amd64.tar.gz
export go_version_path=./tmp/go-version.txt
export go_pkg_remote=https://storage.googleapis.com/golang/go1.7.4.linux-amd64.tar.gz

export splunk_pkg_path=./tmp/splunk-linux-x86_64.tgz
export splunk_version_path=./tmp/splunk-version.txt
export splunk_pkg_remote=https://download.splunk.com/products/splunk/releases/6.5.1/linux/splunk-6.5.1-f74036626f0c-Linux-x86_64.tgz

export common_pkg_path=./packages/common/splunk-linux-x86_64.tgz

if [ -a "${go_pkg_path}" ]; then
echo "Go package already exist, skipping download"
else
echo "Go package doesn't exist, downloading"
wget "${go_pkg_remote}" -O "${go_pkg_path}"
fi
echo "${go_pkg_remote}" > "${go_version_path}"

if [ -a "${splunk_pkg_path}" ]; then
echo "Splunk package already exist, skipping download"
else
echo "Splunk pagage doesn't exist, downloading"
wget "${splunk_pkg_remote}" -O "${splunk_pkg_path}"
fi
echo "${splunk_pkg_remote}" > "${splunk_version_path}"

if [ -a "${common_pkg_path}" ]; then
echo "Common package path already exists, nothing to see here"
fi
echo "${common_pkg_path}" done

echo "Adding blobs"
bosh2 -e vbox add-blob "${go_pkg_path}" golang/go-linux-amd64.tar.gz
bosh2 -e vbox add-blob "${go_version_path}" golang/go-version.txt
bosh2 -e vbox add-blob "${splunk_pkg_path}" splunk/splunk-linux-x86_64.tgz
bosh2 -e vbox add-blob "${splunk_version_path}" splunk/splunk-version.txt
bosh2 -e vbox add-blob "${common_pkg_path}" common/splunk-linux-x86_64.tgz

echo "Creating release"
create_cmd="bosh2 -e vbox create-release --name cf-splunk --tarball="./release.tgz" --force"
if [ "$version" != "" ]; then
create_cmd+=" --version "${version}""
fi

eval ${create_cmd}`

releases check on director
{8:49}~/splunk-firehose-nozzle-release:master ✗ ➭ bosh2 -e vbox releases
Using environment '192.168.50.6' as client 'admin'

Name Version Commit Hash
cf-splunk 0+dev.1 18f3418+

(*) Currently deployed
(+) Uncommitted changes

1 releases

Succeeded
{8:55}~/splunk-firehose-nozzle-release:master ✗ ➭

Thx
Bry

[chenziliang]

May I ask why in your case, deployed via bosh is requied?

[brybinary]

hi chenziliang

sure - the current build process for the clustered splunk estate I work with (index clusters, search head clusters etc) is all via ansible code. As a move to BOSH is currently underway, I want to be able to deploy a clustered splunk env using bosh only (possibly with terraform). I am also in engagement with splunk (vendor) as this is still fairly new (via bosh) afaik.

thx
Bry