Organisation of our component libraries

[Mohammer5]

Through out the existence of ui-core and ui-widgets there's always been a question what should go into which library exactly.

I think we decided on:

• "dumb" components go into ui-core
• "smart" components go into ui-widgets

@HendrikThePendric and I were talking about file input components and we agreed that we need two different components:

1. One that is being used for simple file uploads (like in the Import-Export App)
2. One that's fetching a new id which will then be used to upload the file via the fileResource api

While the first one should go to ui-core, the second one should definitely go to ui-widgets, which feels a bit weird.

To me it would feel a lot more natural if we merged ui-core, ui-widgets and the design-system into one repo and organize components based on their category (atoms, molecules, etc). Then we'd have docs coupled with components and everything in one place.
It doesn't necessarily have to be a monorepo which will result in different npm packages.
As long as we get tree-shaking and dead code elimination working it should be fine to have something like @dhis2/components right?

@amcgee mentioned at one point he wouldn't like it if there were new releases in ui-core when ui-widgets updates (and vice versa), so I think keeping it as one repo/package would work around this issue (as there'd be no nothing-releases).

Upsides

• No more questions where to put what
• No separation between smart/dumb
• Putting docs and components together
• components between categories can be used without having to wait for a release
• no more release chains until a change in ui-core throught ui-widgets can be used in apps that depend on that release

[varl]

Thanks for raising this issue, it highlights a lack of clear communication about high level goals and intent from me. I'll try to write up a longer essay about it on the developer portal soon. I'll try to touch on some important points below. For readability I'll split it into multiple comments.

[varl]

The design-system, ui-core, and ui-widgets are separate by design, and the boundaries between them is a valuable trait of the system that pays off in the long term.

Those three libraries are kept separate because they are three distinct things. On a technical level we work with all off them, and sometimes it feels like we need to work on all of them at once, so the best thing to do is to combine them into one thing.

This is a fallacy of perception, we need to "zoom out" to understand the scope of each individual library and how they fit into the global DHIS2 developer ecosystem.

We have 15 core developers, but there are hundreds to thousands of external developers building DHIS2 apps.

Low-level: Design System

Completely detached from any one implementation, this is the design guidelines for how to implement the DHIS2 look and feel in applications and components. We, or external developers, can use this to build their own ui-core.

Fact is that ui-core is not (and should not be) the only implementation of the design system. A external team has implemented components in Angular based on the design system repo.

Mid-level: UI Core

Our core UI library is the reference implementation of the building blocks in the design system. The components should be usable by any application that satisfies the requirements (React 16.3), no matter if it is a DHIS2 App or something entirely different.

High-level: UI Widgets

This is built on our core components and adds DHIS2 specific business logic into the mix. This might be tight coupling to the DHIS2 API, or that the composed component should behave in a DHIS2 specific way which doesn't make sense for other applications.

This is subtle and important. There needs to be some friction here to flush out what can be general for any application and what is special for a DHIS2 application.

A "widget" can also be a component which requires specific external dependencies to function, where-as our core components should never require additional dependencies to function.

At the "widget" level we are forcing our development practices on developers, so if they want to use our complex components, they need to be on-board with that (e.g. the DataProvider pattern, React Hooks).

If they do not want that, they can drop down to ui-core, where we do not force our way of building applications much more than a HTML component forces it.

---

Annex

Examples of additional "mid-level" libraries

These libraries do not force development practices on developers, they can be used pretty much anywhere. This is the "have it your way" libraries.

• @dhis2/ui-icons
• @dhis2/prop-types

Examples of additional "high-level" libraries

These libraries do force development practices on developers, and can be considered "best practices" of the core developers. This is the "our way or the highway" libs.

• @dhis2/app-runtime
• @dhis2/ui-forms
• @dhis2/analytics

[varl]

@HendrikThePendric and I were talking about file input components and we agreed that we need two different components:

1. One that is being used for simple file uploads (like in the Import-Export App)

2. One that's fetching a new id which will then be used to upload the file via the `fileResource` api

While the first one should go to ui-core, the second one should definitely go to ui-widgets, which feels a bit weird.

I understand where you are coming from based on:

To me it would feel a lot more natural if we merged ui-core, ui-widgets and the design-system into one repo and organize components based on their category (atoms, molecules, etc). Then we'd have docs coupled with components and everything in one place.

When viewed through that lens, there is friction for a developer as she works on "all three repos simultaneously" and has to jump between them.

Viewed through the lens of the high-level goals the idea is that e.g. a FileInput component (no.1) would be in ui-core. It can be used in a DHIS2 app or in a non-DHIS2 app if someone likes how it looks. Nothing special about it. The usage of the component in ui-core might be:

<FileInput id="MyFile" />

To be reasonably used in a DHIS2 application, where we have the requirement of fetching an id from the server before uploading the file (no.2), we would need to provide a widget for it, which would in this simple example just be a wrapper for the FileInput:

<DataQuery query={...}>
   {* skip the error/loading code *}
   <FileInput id={data.id} />
</DataQuery>

Yes, there is a hoop to jump through.

I believe that this additional hoop leads to better and more robust library design. Components stick out like a sore thumb in ui-core when I think about them through the lens of "mid-level vs. high-level" libraries.

[varl]

Mea culpa: inlining the design-system in ui-core

I introduced the idea of "we can pull in the design-system documentation on components into the ui-core documentation, and we can avoid writing it again. Win-win!".

I have spent some time reflecting on that and realised that while it was well intended (DRY!), it was naive, wrong, and has caused more confusion than clarity.

The design system documentation is a different level of documentation than what should be exposed in the UI library's documentation.

At most we should link from the "UI core Button" to the "Design System atom Button".

• The design system docs provides information on the conceptual level: "when" a component should be used and how it should be implemented

• The UI docs provides information on the technical level: "how" is this component works and it's API.

We should remove the ui-core/stories/info directory and write technical documentation for developers using the UI libraries specifically.

[Mohammer5]

@varl Thanks for answering to this so thoroughly!
I definitely see the long term benifits of what you're trying to achieve with this separation.

If you don't mind, I'd like to put a link into the README of ui-core, ui-widgets and ui-forms to this issue until this is documented on the developer portal (to which I'd link once it's there instead).

I think - with your explanation - the issue is closed in regards to where to put what.

---

We should remove the ui-core/stories/info directory and write technical documentation for developers using the UI libraries specifically.

As this falls under organizing as well, I'd like to pick up an older idea we were talking about a few months ago:

1. I'd be nice to turn dhis2/design-system into an npm package so we could integrate the markdown docs (and in the future the .feature files) in other libraries.
2. I'd like to test some plugins that make it a bit more intuitive for those who use storybook as a source of information. I have no idea how this could look like yet, one idea is:
   1. First "story" of a component is the design-system's markdown rendered as a story
   2. Second "story" is the technical documentation of the component
   3. All following stories describe the various states

Storybook is nice to see the various states a component can have, but I don't like it at all in terms of its documentation UX, no one ever clicks on the "Notes" tab. I assume that barely anyone sees that tab!

Here are some candidates:

• react-storybook-addon-chapters (my favorite so far)
• storybook-readme

[Mohammer5]

@varl I've prepared a demo in ui-core:
dhis2/ui-core#387

I've created a Button.chapters.js story file that you can check out

[cooper-joe]

I'd like to add some detail about how components arrive at the different levels referenced by @varl.

Skipping the design stage

We're experiencing some issues with undocumented components growing in scope whilst the build is taking place. Deciding how something should be composed/built cannot be done in a reasonable way until all of the functionality has been decided and described in detail.

An example of how we should be working is the Data-table component. The design documentation has been written before any dev work has started. Now a discussion can take place around the best way to compose the required functionality, what is possible technically, at which library levels the components should be added and so on.

An example of how we should not be working is the FieldSet/ToggleGroup component. The difficulty assigning the correct component scope is due to the lack of design documentation. Without knowing the requirements of the component it is impossible to effectively build it, and attempting to shoehorn UX requirements into an already composed component will lead to sub-optimal user experiences.

Design to Build Workflow

All components, at both mid- and high-level, require design documentation before build can begin. If a component lacks documentation, please raise an issue on JIRA and assign it to Joe Cooper.

Functionality not included in design documentation should not be added on the fly. There will always be more features we could add, but we are intentionally keeping components simple until the functionality is required. We need to avoid adding functionality in a 'we might as well add it while we're here' mentality, otherwise we risk building a bloated and unneccessarily complex system. Potential features that could be added underway could be added as PR's to the Design System, or raised with Joe Cooper informally. However, for functionality to be added to the component it must be documented in the design system and/or build specs. This could be a single sentence describing functionality, or a whole new section in the spec. The important point is that it has been discussed and designed earlier.

This workflow may slow development of new components, but will ensure we are building the right components with the right functionality. Note that this workflow does not mean that whatever is specced/designed is automatically built, but rather that discussion and changes should be made at this stage, not while build is underway.

---

In conclusion: all components added to a ui-* library need design specs before build can begin.