Support macOS Catalina #1032

ghost · 2019-10-15T16:38:20Z

At the moment builds with leveldb feature on macOS Catalina are broken because leveldb-sys crate cannot be compiled. Builds on macOS Mojave still work.

The text was updated successfully, but these errors were encountered:

ghost · 2019-10-16T11:40:24Z

Waiting for skade/leveldb-sys#12 to be merged.

LucioFranco · 2019-10-18T15:50:45Z

@a-rodin last I talked with florian he said that crate is pretty much unmaintained, maybe it makes sense for us to fork for the moment?

LucioFranco · 2019-10-18T15:51:29Z

and I also think this issue is a decently high priority since most mac users will probably switch over soon.

ghost · 2019-10-18T16:39:49Z

@a-rodin last I talked with floian he said that crate is pretty much unmaintained, maybe it makes sense for us to fork for the moment?

I agree that having a fork would make it possible to iterate faster.

I'm not sure is it better to have the fork in a separate repository or in lib in Vector's source tree.

LucioFranco · 2019-10-18T16:43:56Z

Yeah, I want to hear what @lukesteensen has to say. This also leads me to the thing I've probably said a thousand times now maybe now is the time to move to something like sled? (may not be but just thought I'd bring it up)

ghost · 2019-10-18T17:00:14Z

I wonder if we eventually end up with forks of large portion of our dependencies. ClickHouse seems to be doing this, with almost half of contrib dependencies being checked into the tree because of the need for additional patches.

LucioFranco · 2019-10-18T17:03:28Z

@a-rodin I think we can avoid this. It is usually possible to reach out or find people to reach out. We should only vendor things that we know the maintainer will not update. In this case, I had a chat with Florian at RustConf about it and this is pretty much what he said. I think also being apart of the rust open source communtiy I would like for us to focus on giving back as much as possible so trying to get the patches in. We've already done this a lot with tokio and tower.

LucioFranco · 2019-10-18T18:36:57Z

I stand corrected! https://twitter.com/Argorak/status/1185262666652237825?s=20 and happy that I was wrong :)

ghost · 2019-10-21T11:49:42Z

I made a clean installation of Catalina (previously I was just upgrading from Mojave). Now I see the same issue with rdkafka.

I've created a PR to the upstream to fix it: fede1024/rust-rdkafka#160 and #1063 to use our fork in the meantime.

ghost · 2019-10-21T13:43:39Z

Also there are problems with linking to SSL the final Vector executable:

  = note: Undefined symbols for architecture x86_64:
            "_sasl_dispose", referenced from:
                _rd_kafka_sasl_cyrus_close in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
            "_sasl_client_step", referenced from:
                _rd_kafka_sasl_cyrus_recv in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
            "_sasl_client_new", referenced from:
                _rd_kafka_sasl_cyrus_client_new in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
               (maybe you meant: _rd_kafka_sasl_client_new)
            "_sasl_listmech", referenced from:
                _rd_kafka_sasl_cyrus_client_new in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
            "_sasl_errdetail", referenced from:
                _rd_kafka_sasl_cyrus_client_new in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
                _rd_kafka_sasl_cyrus_recv in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
            "_sasl_client_start", referenced from:
                _rd_kafka_sasl_cyrus_client_new in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
            "_sasl_client_init", referenced from:
                _rd_kafka_sasl_cyrus_global_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
            "_CRYPTO_set_locking_callback", referenced from:
                _rd_kafka_transport_ssl_term in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
                _rd_kafka_transport_ssl_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
            "_sasl_errstring", referenced from:
                _rd_kafka_sasl_cyrus_global_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
                _rd_kafka_sasl_cyrus_client_new in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
            "_sasl_getprop", referenced from:
                _rd_kafka_sasl_cyrus_recv in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_sasl_cyrus.o)
            "_CRYPTO_get_locking_callback", referenced from:
                _rd_kafka_transport_ssl_term in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
                _rd_kafka_transport_ssl_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
            "_SSLv23_client_method", referenced from: 
                _rd_kafka_transport_ssl_ctx_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
            "_CRYPTO_num_locks", referenced from:
                _rd_kafka_transport_ssl_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
            "_SSL_load_error_strings", referenced from:
                _rd_kafka_transport_ssl_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
            "_sk_pop_free", referenced from:
                _rd_kafka_transport_ssl_ctx_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
            "_SSL_library_init", referenced from:
                _rd_kafka_transport_ssl_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
            "_OPENSSL_add_all_algorithms_noconf", referenced from:
                _rd_kafka_transport_ssl_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
            "_CRYPTO_set_id_callback", referenced from:
                _rd_kafka_transport_ssl_term in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
                _rd_kafka_transport_ssl_init in librdkafka_sys-21d4ed2b9da07e24.rlib(rdkafka_transport.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

It looks like a conflict between different versions of OpenSSL.

LucioFranco · 2019-10-21T18:35:46Z

@a-rodin ah you need to add a few env vars I think

if openssl is installed via homebrew

export OPENSSL_ROOT_DIR="/usr/local/opt/openssl"
export OPENSSL_LIB_DIR="$OPENSSL_ROOT_DIR/lib"
export OPENSSL_INCLUDE_DIR="$OPENSSL_ROOT_DIR/include"
export CFLAGS="-I$OPENSSL_INCLUDE_DIR"
export LDFLAGS="-L$OPENSSL_LIB_DIR"

stormasm · 2019-10-21T19:33:43Z

@LucioFranco So here is my take on this, for macOS Mojave 10.14.4

Vector used to build out of the box on Mojave prior to this pull request

#927

Now its broken, because the ssl feature was added to rdkafka

rdkafka = { version = "0.20.0", features = ["ssl"], optional = true }

and IMHO Vector is so cool that we should not limit people's experience in building Vector.
As we all know first experience matters...

And we are now REQUIRING people to have openssl installed on their Mac where
as before #927 we did not ??

My work around is simple...

On macOS one has (2) choices...

Either fix is a function of modifying Cargo.toml
I believe the prefered fix is number 1 because rdkafka is still in there,
but if you are not using kafka then number 2 will work fine too...

Remove the SSL feature from Kafka

rdkafka = { version = "0.20.0", features = ["ssl"], optional = true }
Here is the fix :
rdkafka = { version = "0.20.0", optional = true }

Remove rdkafka from the default parameter

default = ["rdkafka", "leveldb", "jemallocator"]
Here is the fix:
default = ["leveldb", "jemallocator"]

Not sure why rdkafka is enabled by default as many people probably never use kafka
but it worked before so I never noticed it inside the default. But when my build broke
the first thing I did was remove rdkafka and everything was fine... Then I started looking
around and realized it wasn't rdkafka but SSL.

ghost · 2019-10-21T20:56:28Z

@LucioFranco It turns out that installing pkg-config allows to avoid setting all these variables manually:

brew install openssl pkg-config

@stormasm Thank you for your input.

Vector had SSL dependency before that pull request too. It was used to, for example, enable HTTPS support for http sink. #927 added ssl feature in order to allow Kafka TLS from #912, which was about to be merged at the time, work with musl.

On macOS one has (2) choices...

Either fix is a function of modifying Cargo.toml
I believe the prefered fix is number 1 because rdkafka is still in there,
but if you are not using kafka then number 2 will work fine too...

Remove the SSL feature from Kafka

rdkafka = { version = "0.20.0", features = ["ssl"], optional = true }
Here is the fix :
rdkafka = { version = "0.20.0", optional = true }

Remove rdkafka from the default parameter

default = ["rdkafka", "leveldb", "jemallocator"]
Here is the fix:
default = ["leveldb", "jemallocator"]

It is also possible to build Vector with disabled rdkafka without modifying Cargo.toml by running

cargo build --no-default-features --features="leveldb jemallocator"

However, I think we need to add requirements for OpenSSL to CONTRIBUTING.md. #1066 does it.

LucioFranco · 2019-10-21T23:17:40Z

@a-rodin ah ok, does it work now for you then?

stormasm · 2019-10-22T03:32:42Z

I should note that I did not have any issues building vector on Sept 10

https://github.com/stormasm/vector

I forked the repo on that date and everything worked at that time...

So, I was curious about this issue some more and I did as you recommended...

brew install openssl pkg-config

Unfortunately, it is still broken if I do not modify Cargo.toml at all...

ty" "-framework" "CoreFoundation" "-framework" "Security" "-lc++" "-lSystem" "-lresolv" "-lc" "-lm"
= note: Undefined symbols for architecture x86_64:
"_CRYPTO_num_locks", referenced from:
_rd_kafka_transport_ssl_init in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
"_SSL_load_error_strings", referenced from:
_rd_kafka_transport_ssl_init in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
"_sk_pop_free", referenced from:
_rd_kafka_transport_ssl_ctx_init in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
"_SSL_library_init", referenced from:
_rd_kafka_transport_ssl_init in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
"_SSLv23_client_method", referenced from:
_rd_kafka_transport_ssl_ctx_init in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
"_OPENSSL_add_all_algorithms_noconf", referenced from:
_rd_kafka_transport_ssl_init in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
"_CRYPTO_set_id_callback", referenced from:
_rd_kafka_transport_ssl_term in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
_rd_kafka_transport_ssl_init in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
"_CRYPTO_set_locking_callback", referenced from:
_rd_kafka_transport_ssl_term in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
_rd_kafka_transport_ssl_init in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
"_CRYPTO_cleanup_all_ex_data", referenced from:
_rd_kafka_transport_ssl_term in librdkafka_sys-dd31936df6cd8703.rlib(rdkafka_transport.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error

error: Could not compile vector.

stormasm · 2019-10-22T03:35:11Z

After doing the brew install above I added these lines to my .profile

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"

export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

Also note that I installed this as well via homebrew without any issues...

https://formulae.brew.sh/formula/librdkafka

ghost · 2019-10-23T14:12:06Z

@LucioFranco

@a-rodin ah ok, does it work now for you then?

It worked for me, but more by accident. Unfortunately in an arbitrary configuration we still need to export OPENSSL_INCLUDE_DIR and OPENSSL_LIB_DIR because OpenSSL is used in two places, openssl-sys and rkdafka-sys, and these two places look for SSL in a slightly different way. #1074 is related to this.

stormasm · 2019-10-23T15:40:04Z

@LucioFranco @a-rodin thanks for looking into this further...

I will be happy to test your solution on my mac once it gets worked out...

No hurry on this end,
as I am not using the ssl feature on kafka and I just change the config to this...

rdkafka = { version = "0.20.0", optional = true }

Plus, working on the bigger picture re: #1074 sounds like a good place
to be putting more time and energy...

It will also give me the opportunity to better understand native dependencies management

ghost · 2019-11-11T14:33:26Z

@stormasm You can try current master with #1170 merged, which enabled openssl/vendored feature by default. So currently Vector can be built on macOS with only rustup and XCode Command Line tools installed.

stormasm · 2019-11-12T02:59:19Z

@a-rodin Just sent you a file to your email with the whole stack trace for....

macOS Mojave
version 10.14.6

Compiling rdkafka-sys v1.2.1 (https://github.com/timberio/rust-rdkafka#06ddd48d)
error: failed to run custom build command for rdkafka-sys v1.2.1 (https://github.com/timberio/rust-rdkafka#06ddd48d)

Caused by:
process didn't exit successfully: /tmp31/vector/target/debug/build/rdkafka-sys-9916160cd6d2d587/build-script-build (exit code: 101)

ghost · 2019-11-12T15:47:54Z

@stormasm From the logs it looks like you have non-empty environment variables CFLAGS and LDFLAGS (probably the same as the ones mentioned in #1032 (comment)), which interfere with the build system configured to use vendored headers and libraries.

Please try to unset CFLAGS and LDFLAGS and run cargo clean before the build.

stormasm · 2019-11-12T21:20:14Z

@a-rodin Thanks that fixed the problem.
Everything is now up and running and builds out of the box on

macOS Mojave
version 10.14.6

I ran all of the tests and they passed.
along with lots of my own tests and examples which run as well.
thanks for fixing this...

ghost added domain: operations type: task Generic non-code related tasks labels Oct 15, 2019

ghost added this to the Support more targets & improve existing ones milestone Oct 15, 2019

ghost added needs: reply needs: outside help Needs help outside of the Vector core team and removed needs: reply labels Oct 16, 2019

This was referenced Oct 18, 2019

(deleted) skade/leveldb-sys#13

Closed

fix: Fix build on macOS Catalina by explicitly invoking the shell vectordotdev/leveldb-sys#1

Merged

chore(operations): Update leveldb-sys up to 2.0.5 #1055

Merged

ghost closed this as completed in #1055 Oct 18, 2019

ghost reopened this Oct 21, 2019

This was referenced Oct 21, 2019

Fix build on macOS Catalina vectordotdev/rust-rdkafka#1

Merged

fix(operations): Use rdkafka crate from the upstream Git repository #1063

Merged

ghost removed the needs: outside help Needs help outside of the Vector core team label Oct 22, 2019

ghost mentioned this issue Nov 11, 2019

enhancement(operations): Use vendored OpenSSL #1170

Merged

ghost closed this as completed Nov 14, 2019

This issue was closed.

Support macOS Catalina #1032

Support macOS Catalina #1032

Comments

ghost commented Oct 15, 2019

ghost commented Oct 16, 2019

Uh oh!

LucioFranco commented Oct 18, 2019

Uh oh!

LucioFranco commented Oct 18, 2019

Uh oh!

ghost commented Oct 18, 2019 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LucioFranco commented Oct 18, 2019

Uh oh!

ghost commented Oct 18, 2019

Uh oh!

LucioFranco commented Oct 18, 2019

Uh oh!

LucioFranco commented Oct 18, 2019

Uh oh!

ghost commented Oct 21, 2019 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Oct 21, 2019 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LucioFranco commented Oct 21, 2019

Uh oh!

stormasm commented Oct 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Oct 21, 2019

Uh oh!

LucioFranco commented Oct 21, 2019

Uh oh!

stormasm commented Oct 22, 2019

Uh oh!

stormasm commented Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Oct 23, 2019

Uh oh!

stormasm commented Oct 23, 2019

Uh oh!

ghost commented Nov 11, 2019

Uh oh!

stormasm commented Nov 12, 2019

Uh oh!

ghost commented Nov 12, 2019

Uh oh!

stormasm commented Nov 12, 2019

Uh oh!

ghost commented Oct 18, 2019 •

edited by ghost

Loading

ghost commented Oct 21, 2019 •

edited by ghost

Loading

ghost commented Oct 21, 2019 •

edited by ghost

Loading

stormasm commented Oct 21, 2019 •

edited

Loading

stormasm commented Oct 22, 2019 •

edited

Loading