Scala Center Proposal: Community Build

I was chatting on the the scala/community-builds gitter about community builds and the scala center(SC). This particular link was regarding a scala.js community build.

So this issue it about getting more SC participation in this area. Right now, I'm not totally sure what that should be specifically - but I think there is a natural fit here.

But I'll make a start by saying that the scala community build is really for the benefit of scalac - fair enough, it is theirs. I can see why scala.js would want one too.

But many others projects could benefit from their own, too (eg typelevel) . So perhaps a goal of the SC could be to promote the development and use of community builds in general - and this would also nicely tie in with their scaladex.

cc/ @sjrd and @SethTisue as they were on the chat

[tpolecat]

Can someone with sufficient sparkles create a Scala Center label?

[milessabin]

✨

[larsrh]

Agreed in principle. But in order to turn this into a Scala Center proposal we would need to specify some actionable items. Does anyone have something in mind? Off the top of my head: spend some work to make it easy for third parties to curate their own community build. Can we go from there?

A few tweaks to catalysts-docker might well be enough to get the practical side going. The documentation is clearly missing there, but catalysts-docker + dbuild should be enough to get local community builds(CB) going.

The bigger problems stem from when one actually starts to run the CB - esp cyclic/missing dependencies. I'll add a post here later on that....

@SethTisue Don't suppose you know anyone that has any thoughts on the practical aspects of running a community build?

Continuing from earlier:

A few tweaks to catalysts-docker might well be enough to get the practical side going.

I think that once the concept is promoted a bit and a few more people actually start doing such builds, a raft of other issues will become apparent, in particular at the "low-end" of the stack where any problems also have the most impact.

By example (almost real..):

Cats is a library that many libs depend on, and it depends on scalatest (and others). scalatest would like to use cats, but can't, because of the cyclic dependency .

But hey... the cats dependency is only a test dependency on scalatest so it's OK. Ahh, but how do we release cats without testing it? And if we hack it, how do we tell dbuild to hack it?

So one day, someone splits cats into like 12 repositories and re-configures dbuild. Sametime, half the cats maintainers leave as they can no longer work in an environment that is geared towards distributive builds rather than abstract coding.

So the twelve repositories are merged back, into two repos.

And scalatest copies and pastes the bits of cats it needs into scalatest to avoid it's dependency issue, and then....
scala 2.12.0-RC2 is released and then....

I think the real value that the SC can add (in the short term) is the promotion of community builds and collecting the common issues, that will arise from many projects

Given that it looks like the SC are currently "stretched", having a proposal that is more community centric rather than, say, coding centric is useful as :

• the SC are less unlikely to park it.
• the board are less unlikely to reject/park it
• the community are more likely to accept that the SC is moving to encompass them

Also, most initial work would likely come from scalac, scala.js, typelevel - hence allowing the SC to really focus on the soft part.

[sjrd]

scalatest would like to use cats, but can't, because of the cyclic dependency .

IMO the story simply stops there. It can't, period. Do not introduce cyclic dependencies, even if it's "just" in your Test config.

That's why Scala.js uses JUnit and not ScalaTest or any other Scala dependency. Doing anything else is madness.

[SethTisue]

Don't suppose you know anyone that has any thoughts on the practical aspects of running a community build

what would you like to know? I can certainly confirm there are "practical aspects". it's the single thing I've spent the most time on.

I recently did a pass through the documentation at https://github.com/scala/community-builds/wiki and updated everything, so that's a good source of information.

Jenkins requires care and feeding (https://github.com/scala/scala-jenkins-infra). somebody has to pay the AWS bills, too (thx, corporate overlords ✨). the interaction with PR validation on GitHub (https://github.scom/scala/scabot) takes care and feeding. all of this stuff is on GitHub and available for raiding, but adapting or repurposing it for other orgs needs besides just the Scala compiler team's would be work, of course.

dbuild was a great deal of work to create (thx again 💵✨). but you should know that there is zero or nearly-zero budget for ongoing development, at least at the moment. the Scala community build uses dbuild 0.9.5, it works fine, if we found a bad bug probably @cunei would find some spare moments to fix it for us, but in your planning, you might want to take into account that you can't assume that dbuild is actively being developed and improved. it isn't. Lightbend used to use it to build the Lightbend Reactive Platform, but we don't anymore as of RP 2, so that's no longer a driver.

as far as whether this ought to be a Scala Center priority: I like the Scala.js-community-build idea. I'm not sure there are other problems this solves?

[SethTisue]

here is a concrete suggestion for Typelevel: we would have love to have more Typelevel projects in the Scala community build. I've made some progress on it in the last year-plus. but adding them is work. quite apart from whether y'all decide to do your own community-build-like-thing, It would be fantastic if the Scala community build itself had higher visibility within Typelevel and if it were encouraged and considered normal for library authors to add their libraries to the community build. I've done my best to document what's involved. it's usually necessary for me to do some handholding since library authors aren't familiar with dbuild, etc, but even with handholding, it's a lot less work for me than doing the whole thing myself.

we're no longer accepting additions to the 2.11 community build; here is the 2.12 projects list: https://github.com/scala/community-builds/blob/2.12.x/project-refs.conf

why is this beneficial for Typelevel, and not just beneficial for the Scala team? well, we'd like to think that anything that helps us helps y'all as well. but also, your projects push scalac pretty hard. we do our best not to break your stuff, but, also we can't always know in advance whether changes we're making might adversely impact your projects. if your code is in the community build, we know right away (sometimes even before a Scala PR is merged — we don't run the community build against every PR, but we run it against the tricky ones). so it's insurance for you that the things you're doing with Scala will continue working

[SethTisue]

Do not introduce cyclic dependencies

+1. dealing with the cyclic dependencies between scala and scala-xml, and scala and scala-parser-combinators (fixed in 2.12), and scala and scala-partest, and scala and scalacheck, consumes an inordinate amount of thought and effort. it's so much better not to get into this kind of trouble in the first place

IMO the story simply stops there. It can't, period. Do not introduce cyclic dependencies, even if it's "just" in your Test config.

Yep, that's the point!! I know that, you know that.. but the question is - where are the cyclic dependencies in the whole OSS community?. I think we know where they are at the "bottom" and I recently did a PR for new scoverage core that just uses junit.

But I did that because "I knew there was an issue" - but technically, someone else, or a process, could have done so. And the same process could work on many projects, for example from scaladex

Doing anything else is madness.

Another 👍 I'm sure there is plenty of madness in the scala OSS community as a whole, and that is where perhaps the SC can help: not to solve the issues, but highlight them somehow (for the more popular projects)