Create - STATE

[alyhxn]

@todo

#19

---

• Create - STATE
  • Implement base
  • Integrate into components
  • Providing secure data to components
  • Listen for changes and trigger relevant response
  • Store and access all data from DB
  • Hanlde project_name while using fetch for github
  • Handle $ref localization for overrides.
  • Improvements to better support data operations
  • Finalize
  • @output ❓ STATE_v0.0.?
    • next Create - theme_widget #21

[alyhxn]

Tasks - 2024.10.18

• Update - STATE_v0.0.1
  • Implement the fallback management - 4h
  • Planning and discussions - 15h
  • Debugging - 2h
  • Logged tasks - 5min
  • Recoreded Worklog - 10min
  • @output 📦 STATE_v0.0.2

---

Worklog

worklog-238

[alyhxn]

Tasks - 2024.10.24

• Update - STATE_v0.0.2
  • Updated fallback data structure - 1h
  • Changes due to change in data structure - 3h
  • Planning and discussion - 5h
  • Implementing level-1 override - 2h
  • Authored docs for fallbacks - 3h
  • Logged tasks - 5min
  • Recorded Worklog - 10min
  • @output 📦 STATE_v0.0.3
  • @output 📦 Fallback_docs

---

Worklog

worklog-239

[serapath]

feedback 2024.10.24

for worklog 238 and 239

First, I feel the worklog video is missing a lot of things.

1. It does not have a section about DB and you posted worklog238 on that issue too
2. It does not have a section about parser and you posted worklog238 there too (Refactor - playproject #19 and Create - DB #31)
3. It does not have a section to walk through the doc/... updates, but fallback docs are linked as an output.

I would be happy if you could structure future worklogs to cover all the outputs and changes you worked on. Either one after another, or if it doesnt fit into 5 minutes, then multiple worklog videos.

Having said that, here my FEEDBACK: 😺

STATE.js

STATE.js should not require a hard coded snapshot.json file

status.module_index = {}
status.fallback_handlers = {}

graph_explorer

+1 for highlight color for currently focused entry
+1 for color highlighting duplicate names, it is a bit much though

• could we just give duplicates a "pale grey font color"?
• the original entry should not have any special color
• when any pale grey duplicate is selected, it shows the highlight color for
  • the original entry (shortest path from root to entry)
  • and all its expanded duplicates

+1 for showing state when component instance is selected
+1 for showing javascript source code files

snapshot.json

Only in the theme widget, there should be special action buttons to:

• export/dump the entire database into a snapshot.json file to download
• import and populate the database from a user provided snapshot.json file

There needs to be a special admin... method to allow such an operation

require

The require function will be customized by us, but we have to implement it all without that.
Basically, once we patch the require function, we wont need to require the STATE module anymore. The lines

const STATE = requuire('STATE')
const statedb = STATE(__filename)

will be automatically prefixed into every module.

Apart from that, the main plan is to get rid of budo, but thats it for now.

fallback

yes, fallback system is really simple in its core idea, but quite challenging and complex to implement for sure.

It is confusing that this fallback is called index.
Shouldnt there be a proper module name instead?
What is the file required? is it just "."? In that case, let that be the name instead of index.

I agree that idx:1 doesnt tell anything, neither does type:1, but then again,
all it takes is scrolling in the same file, because fallback_instance and fallback_module
are both part of the same file, so it should be easy to figure out.
Also usually, the "default fallback data" does not include hundreds of instances or modules, but even then you can still figure it out. All it needs is a CTRL+F and seach for e.g. 1: and it will probably quickly find the key of the required module entry to say type: '...' and you know.

We could of course add the name of the module, but it makes everything more verbose as well.
I'm not convinced it improves anything. I feel unsure about idx though looking at it.

When i see idx, i would assume it is an index into an array or something.
I constantly have to remind myself, that this is the "module type" of that specific instance.

This seems fine.
A fallback override for a direct sub component makes sense.

Just for the sake and in case we dont have another good example, those
topnav entries could be a "button instance", so that we get the chance to try out
more complex fallback structures, but maybe we get that with contributors or projects already.

QUESTION:
I am watching the files and the video and every time you open e.g. a module.json or instance.json, i need to always think for which module or component is it.
If I'm inside a file and I watch the code, the fallback function gives me no insights,
apart from e.g. require('module.json').
Now i get it, if the instance of module fallback data is large, maybe we want the ability to
have a separate file for it, BUT - if it is as little data as in all your examples,
inlining the data makes it so much easier to quickly read what is going on imho.

That fallback is a function and doesn't really care how you structure your code, apart from
you having to return the "default data" is clear, so people can be creative how they use it.
I think for us having the data inside the code file itself is quicker for now.

Do you agree? And if so, could we try to change it and inline all fallback data and skip external fallback data like "module.json" or "instance.json" for now? :-)

prefix (153)
We need that in the localStorage.
Think of it as a process tree id.
We could technically have an option later to restart the entire app a second time
Or to start and modify the app under lets say 154.
We could then switch processes between 153 and 154 - both using different database data.
In order to be able to play with that, we need all data stored in such a way that it is it's own "process tree".

[alyhxn]

Tasks - 2024.10.28

• Update - STATE_v0.0.3
  • Removed populate and added fetch_save - 15min
  • xtype removed from data - 6h
  • Now we are using module's ID instead of name, not using type + idx - 1h
  • Batch is now array - 10min
  • Version code updated - 1min
  • Code updated with replaceChildren - 1min
  • Updated subs as mentioned - 25min
  • Moved the content file to web - 1min
  • Updated the docs a bit - 30min
  • Moved default data inside the module - 30min
  • Implemented multi-level fallback - 5h
  • Discussion - 3h
  • Cleaned STATE.js code - 3h
  • Logged tasks - 10min
  • Recorded Worklog - 40min
  • @output 📦 STATE_v0.0.4

---

Worklog

worklog-240
worklog-240.2
worklog-240.3

[alyhxn]

Tasks - 2024.11.02

• Update - STATE_v0.0.3
  • Unified module and instance fallbacks - 2h
  • Updated code accordingly - 2h
  • Logged tasks - 5min
  • @output 📦 STATE_v0.0.4

Feedback

• Implemented the code as asked so not much to explain in a video.

[alyhxn]

Tasks - 2024.11.07

• Update - STATE_v0.0.4
  • Code snippet - 2h
  • Discussion - 3h
  • Updated footer.js and header.js with the latest code fallback DS - 3h
  • Logged tasks - 5min
  • Recorded Worklog - 10min
  • @output 📦 STATE_v0.0.5

---

Worklog

worklog-242.2

[serapath]

feedback 2024.11.08

Lets compare our current and potentially new approach
Given: demo > app > foo > head > nav > menu > (btn | btm.small) > icon

1

// ----------------------------------------------------------------------------
// 1. make icon set its default fallback (=`FB_II`) using the `image.svg` in the above snippet
// icon.js
/***********
  NOW:
***********/
function FB_II () {
  return { 'image.svg': `<svg>▶️</svg>` }
}
/***********
  vs. NEW:
***********/
function FB_II () {
  return { 'image.svg': `<svg>▶️</svg>` }
}

This is the same, no changes.

---

2

// ----------------------------------------------------------------------------
// 3. make button override icon `image.svg`
// 2. set button default fallback (=`FB_IB1` + `FB_IB2`) to `label/size`
// btn.js
/***********
  NOW:
***********/
function FB_IB () {
  return {
    0: { subs: [1], data: { label: 'button', size: 'small' } },
    1: { type: 'icon', override: [override_icon] }
  }
}
function override_icon (data) {
  data[0].data['image.svg'] = `<svg>🧸</svg>`
  return data
}
/***********
  vs. NEW:
***********/
function FB_IB1 () {
  return {
    _: { icon: { 0: override } }, // icon === 'icon:0'
    'data.json': { label: 'button', size: 'large' }
  }
}
function FB_IB2 () {
  return {
    _: { icon: { 0: override } }, // icon === 'icon:0'
    'data.json': { label: 'button', size: 'small' }
  }
}
function override (data) {
  data['image.svg'] = `<svg>🧸</svg>`
  return data
}

This is different.

1. the override data[0].data['image.svg'] = seems more verbose in current state
2. your version does not overridethe normal "large" button, only the small one.
3. the override: [...] array needs a user to know the paths of sub components to be able to fill it out with more than just a single function, so they have to know the source code

---

3

// ----------------------------------------------------------------------------
// 5. make menu code require 2 button module instances, one for small button, one for normal button
// 4. make menu (=`FB_IM`) override button label + reset icon back to original from what button changed
// menu.js
/***********
  NOW:
***********/
function FB_MM () {
  return { 0: { subs: ['btn'] } }
}
function FB_IM () {
  return {
    0: { subs: ['1', '2'] },
    '1': { type: 'btn:small', override: [override_btn] },
    '2': { type: 'btn:normal', override: [override_btn] }
  }
}
function override_btn (data) {
  data.label = 'beep boop'
  Object.entries(data).forEach(([id, value]) => {
    if (value.type.includes('icon')) data[id].override = null
  })
}
/***********
  vs. NEW:
***********/
// FB_MM
function FB_IM () {
  return { _: { btn: { 0: override }, 'btn:1#small': { 0: override } } } // btn === 'btn:0'
}
function override (data) {
  data['data.json'].label = 'beep boop'
  data._['icon:0/0'] = ([icon]) => icon()
  return data
}

1. the FB_IM in the new version is much smaller
2. the override does not need to test for type 'icon', because its in the key
   • again it is also smaller in code
3. data.label next to data[id] seems weird, because in button:
   • the label is part of the data[id=0] attribute, so this is confusing
4. the override: [...] array needs a user to know the paths of sub components to be able to fill it out with more than just a single function, so they have to know the source code

---

4

// ----------------------------------------------------------------------------
// app.js
// foo.js
// head.js
// ----------------------------------------------------------------------------
// 6. make nav (=`FB_IN`) undo the menu overide for button module instances
// nav.js
/***********
  NOW:
***********/
function FB_IN () {
  return {
    0: { subs: ['1'] },
    '1': { type: 'menu', override: [override_menu] }
  }
}
function override_menu (data) {
  Object.entries(data).forEach(([id, value]) => {
    if (value.type.includes('btn')) data[id].override = null
  })
  return data
}
/***********
  vs. NEW:
***********/
function FB_IN () {
  return { _: { menu: { 0: override } } } // menu === 'menu:0'
}
function override (data) {
  data._['btn:0/0'] = ([btn]) => btn()
  data._['btn:1#small/0'] = ([btn]) => btn()
  return data
}

1. again, FB_IN in new version is shorter
2. override in new version does not need to loop and check for type
3. also your check might fail if there is a module called "debtnote"
   • maybe a component card to visualize the debt of a user?
   • so more a heuristic than a very exact way to do it
4. the override: [...] array needs a user to know the paths of sub components to be able to fill it out with more than just a single function, so they have to know the source code

---

5

// ----------------------------------------------------------------------------
// 7. make demo (=`FB_MD`) redo the menu override
// demo.js
/***********
  NOW:
***********/
function FB_MD () {
  return {
    0: { subs: ['app/foo/head/nav/0'] },
    // PROBLEM_1: this does **not** work: '0' key will overwrite 0 key
    '0': { type: 'menu', override: [override_menu] }
  }
}
function override_menu (data) {
  // PROBLEM_2: how does this "redo" the original "menu override" from menu default data?
  data.menu.items.push('demo')
  return data
}
/***********
  vs. NEW:
***********/
function FB_MD () {
  return { _: { app: { 0: override } } } // app === 'app:0'
}
function override (data) {
  data._['foo/head/nav/menu/0'] = ([menu]) => menu()
  return data
}

There seem to be some issue with your version:

1. see PROBLEM_1
2. see PROBLEM_2
3. how do you KNOW that app/foo/head/nav/0 exists without looking at source code?
   • the NEW version can console.log(data._) to learn about available paths
4. The NEW version has a much shorter FB_MD function
5. the override: [...] array needs a user to know the paths of sub components to be able to fill it out with more than just a single function, so they have to know the source code

[alyhxn]

Tasks - 2024.11.13

• Update - STATE_v0.0.5
  • Converted fallback DS to tree. 1h
  • Made relevant changes in the code 3h
  • Constructed the tree to show the path - 3h
  • Update fallbacks/override with new proposal - 2h
  • Logged tasks - 5min
  • Recored worklog - 5min
  • @output 📦 STATE_v0.0.6

Worklog

worklog-243
worklog-243.1

[alyhxn]

Tasks - 2024.11.20

• Update - STATE_v0.0.6
  • Discussion for creating multiple modules of the same type - 3h
  • Experiment - 5h
  • Created new repo - 30min
  • Implemented example with the boot.js solution - 1h
  • Debugging the hidden issues - 2h
  • Logged tasks - 5min
  • Recorded Worklog - 20min
  • @output 📦 STATE_v0.0.7

Worklog

worklog-244
worklog-244.1

[serapath]

feedback 2024.11.20

1. regarding const default_slots = ['hubs', '_', 'inputs', 'outputs'], developers using the graph-explorer are supposed to define slots, but the STATE module is not supposed to have custom slots, apart maybe from the data defined in fallbacks and the "dataset groups/mapping/switching" we havent talked about yet.

2. you havent refactored yet, but just to comment on some things:

   • 
   • In my opinion this screams for a short helper function to get the module name from a filepath or whatever. The exact splitting and extracting of relevant string parts can be details of that function. That way, i can read the function call name and know what it does and if interested in details, i can go to that function and check how it does it.

3. another thing that is confusing is localdb and the api. i expected a simple key value store

   • 
   • where is data defined? i cant find any var/const/let data = ...
   • why do we need those "search_filters"? you say because modules dont pass sid down, but:
     1. we are supposed to know the require path, e.g. app/head/menu/button
     2. when require('button') runs we can count how often we see app/head/menu/button
     3. so we can count and based on execution order make it button:0 and then button:1
     4. this is already true for super modules, thus: app:0/head:0/menu:0/button:0
     5. this can serve as unique ID and make passing a sid irrelevant for modules

4. i dont get status.fallback_check so i assume that when it is true, then the database is empty and we execute fallbacks? ...otherwise i am confused, because we execeute the fallback and preprocess when the if check is true... so ehh..yeah. let me know.

[serapath]

feedback 2024.11.20

Looks good. I have no opinion yet on whether name:x or name#x or name:x:y. I agree we need some convention. If yours work, lets roll with it for now and lets make everything work and feature complete. Once it works, we can do another iteration about changing names or symbol conventions, etc...

I didnt understand what or why we need something extra for export/import, ...like a $ sign, but if there is a need, lets use it. Any sign is okay for now, lets just minimize what we need to make all features work and then we take another look.

I am not yet super happy with the path argument for override functions. I was more thinking those would already be part of the data returned by e.g. topnav() or any other function provided as an argument ...e.g. [topnav, menu, button] if we try to override button in head of the app that has head/topnav/menu/button.
I have not understood yet what the advantage of path is or why we need to have it like that.

---

issue: topnav#1.json

I think this is a slight misunderstanding.
The idea is fallbacks/overrides execute when the database is empty, so no files exist yet in the database.

Now when topnav() executes, it is supposed to create a "fresh instance" of data.inputs['topnav.json'] which is distinct from the one in topnav#1 even though it has the same name.

Imagine EVERY COMPONENT HAS ITS OWN VIRTUAL DISK :-)

So e.g.

• localStorage['app#0/head#0/topnav#0'] = {}
• localStorage['app#0/head#0/topnav#1'] = {}
  They will both contain a key called topnav.json: ... with the content and if one components edits their version, it should not change the other component.

In fact, every single "component instance" will also have ITS OWN VIRTUAL DISK too :-)

Now IF we want to have many component "sharing" a file, then

1. now a module developer can give all instances of that module access to the same data that belongs to the module, not to any specific instance
2. otherwise, if multiple instances should share a different file, that is not in the data of the module they belong to or where instantiated from, then there needs to be a super component instance to override the defaults of multiple different instances with a pointer to the same file, which can be a file stored in that super component for example.

I guess based on use cases - requesting and sharing files or data between multiple components is something we have to work on, but for now, by default, everything has to have its own file copies.

This means, the components:

• app:0/head:0/topnav:0/menu:0/button:0
  • has app:0/head:0/topnav:0/menu:0/button:0#button.json
• app:0/head:0/topnav:0/menu:0/button:1
  • has app:0/head:0/topnav:0/menu:0/button:1#button.json

where app:0/head:0/topnav:0/menu:0/button:0 is the id of the button instance 0 and also the id of the "virtual disk" and e.g. #button.json is a file stored in that disk. This is also how the database can be structured.

Now the above is not yet meant to make a decision how we use: /, :, # or maybe $ or other kinds of separators to structure those name spaces... lets use something and later, when everything works, we can see what seems best.

Looking forward for multiple named exports and what notation you choose there :-)

---

regarding fallback data:

• JSON.parse(JSON.stringify(...))
• or require('./instance.json') with `delete require.cache[require.resolve('./instance.json')]

The fastest and most appropriate way to do this these days is ti use https://developer.mozilla.org/en-US/docs/Web/API/Window/structuredClone

BUT ...in the case you mentioned, i would highly recommend to maybe just define the entire fallback in a module instead, like:

// instance.json.js
module.exports = () => {
  return {
    // content of ./instance.json
  }
}

instead of

Because that means for every instance we can just call instance() to get a fresh copy of the data.

I personally would probably just define the fallback functions at the end of the current module file and keep it there instead of an external file, but the good thing with our system is:

=> People can do it in their module however they want and people programming other modules dont need to really care and things still work :-) ...great that there are so many ways to do it :P
Thats a feature.

---

Regarding chatgpt - i strongly feel, that neither for our STATE module, nor for our fallbacks, nor for the details about the require function and many more things, like the stack trace parsting we tried to do and plenty more to come - will be properly answered by chatgpt.

If it helps you, that's great, but my past experiment is, that apart from basic stuff, it fails and you are then quite busy fixing all the mistakes that in the end you didnt even save time at all.
Chatgpt seems to be only good in solving generic beginner problems in a helpful manner, but yeah...

Also openAI already announced that progress isnt that fast anymore and they hit a limit where progress slows down significantly, so i expect no change in the situation any time soon.

well - kinda irrelevant feedback. If it feels helpful to you, go ahead - whatever works :P

It might have told you about structuredClone perhaps :D

[alyhxn]

Tasks - 2024.11.25

• Update - STATE_v0.0.7
  • Worked on feedback - 3h
  • Added example modules - 5h
  • Debugging - 2h
  • Logged tasks - 5min
  • @output 📦 STATE_v0.0.8

Worklog

worklog-245
worklog-245.1

Feedback

Oh, I forgot to talk about the graph explorer. It has been updated as proposed. Clicking the line expands top and clicking the emoji expands bottom.

[serapath]

feedback 2024.11.24

---

regarding slots

you answered, but i'm unclear why we need slots at all in the STATE module. I have a feeling we dont.

---

regarding .find and search_filters

for modules we dont know the sid, but, we can know the path and the order in which they get required, e.g.

1. /page/app/header/topnav/menu/button/icon could be the location to require the "icon module".
2. When we detect that order inside of a call to statedb(...), we can:

const modules = {}
// ...
module.exports = STATE
function STATE (__filename, modulepath) {
  const prefix = modulepath.join('/')
  return statedb
  function statedb (...fallbacks) {
    const m = modules[prefix] || (modules[prefix] = { counter: 0 })  const id = `${prefix}:${m.counter++}`
    const sid = Symbol(id)
    // ...
  }
}

This way we can just NOT use an id, but instead a modulepath to identify a specific module instance.
The "modulepath" is unique per module, but if 2 instances are required in the same modulepath, we can count the order.

Also, the modulepath is NOT the __filename, because each segment of that modulepath has its own filename. In order to track that we might need to modify the reuqire function, but luckily we now have boot.js where we can exactly do that and also pass a unique require function to each module, which means, we can patch it like this:

function patch_cache_in_browser (source_cache, module_cache) {
  const meta = { modulepath: [], paths: {} }
  for (const key of Object.keys(source_cache)) {
    const [module, names] = source_cache[key]
    const dependencies = names || {}
    source_cache[key][0] = patch(module, dependencies, meta)
  }
  function patch (module, dependencies, meta) {
    const MAP = {}
    for (const [name, number] of Object.entries(dependencies)) MAP[name] = number
    return (...args) => {
      const original = args[0]
      require.cache = module_cache
      require.resolve = resolve
      args[0] = require
      return module(...args)
      function require (name) {
        const identifier = resolve(name)
        if (name.endsWith('node_modules/STATE')) {
          const modulepath = meta.modulepath.join('/')
          const original_export = require.cache[identifier] || (require.cache[identifier] = original(name))
          const exports = (...args) => original_export(...args, modulepath)
          return exports
        } else if (require.cache[identifier]) return require.cache[identifier]
        else {
          const counter = meta.modulepath.concat(name).join('/')
          if (!meta.paths[counter]) meta.paths[counter] = 0
          const localid = `${name}:${meta.paths[counter]++}`
          meta.modulepath.push(localid)
        }
        const exports = require.cache[identifier] = original(name)
        if (!name.endsWith('node_modules/STATE')) meta.modulepath.pop(name)
        return exports
      }
    }
    function resolve (name) { return MAP[name] }
  }
}

Using the above as a strategy, we can skip the searching i'd say. What do you think?

---

regarding the path argument

I do think your analysis that the data returned from an override function would potentially contain all data down to the leaf nodes. Would that be different with using path though?

And even though the data would contain data down to the leaf node, we dont have to execute or apply it yet. We can store it in a cache inside the STATE module until we actually need it and then we apply it.

Maybe you can explain a bit more in depths how it would work with path instead. Is there an example? I dont yet understand it.

---

example.html

can we try to mimic exactly the structure defined in the discord chat? I like the way you made it work, but i would prefer the following structure, so we can have more edge cases.

/* page structure:
 - page:0
   - app:0
     - head:0
       - (foo) // does NOT call `statedb(...)`
         - nav:0
           - menu:0
             - btn:0
               - icon:0
             - btn:1#small
               - icon:0
     - foot:0
       - text:0
*/

Of course, it makes sense to keep the extra css and html to make the example look like something :-)
The idea with the hover vs click menu is pretty cool though.

So maybe you can add a second:

           - menu:1#hover
             - btn:0
               - icon:0
             - btn:1#small
               - icon:0

So if you could extend the whole thing a bit, that would help, because i am coding against the same example already for some additional features that we need to integrate it with the sandboxed runtime.

But also - after watching your worklogs, i have not seen the kinds of overrides defined in the example on discord. Could you do that? Because thats the whole point, to see if the overrides really work :-)

Finally, when the whole page is initialized, can you inside of STATE module make sure to define, e.g. window.STATEMODULE or something like that, so i can open the devtools and console.log(window.STATEMODULE) which contains all relevant state thta is being built up by the STATE module when initializing and executing the entire page?

That way i could explore the structure that keeps track of everything and explore it and compare it after the page has successfully loaded, e.g. all the different instances and "virtual file systems", etc...

[alyhxn]

Tasks - 2024.11.30

• Update - STATE_v0.0.8
  • Integrated modulepath - 1h
  • Created example components - 2h
  • Merging data and path - 7h
  • Logged tasks - 5min
  • Recorded worklog - 10min
  • @output 📦 STATE_v0.0.9

Worklog

worklog-246

[serapath]

feedback 2024.12.01

slots

Okay, lets ignore slots for now. I think we need to revist them when we finally get to "dataset group mapping and switching" :-)

---

find/search

I didnt entirely understand how this works.
If you only pass in modulepath, then how do you load the right state from the database? each module instance (e.g. 0, 1, 2...) if it gets required multiple times, will have a different state.

In my mind, this can become a simple lookup, where type or path is basically a "unique instance id" and can serve as a key in a key value store to lookup associated data state to be loaded when a module initializes with a non-empty database

---

data/path

you said it is merged.
i still would love to see fallbacks as shown in the discord snippet where page overrides a sub sub sub dependency, e.g. like page/app/head/menu/btn/icon or something like that.

How does it work and how does it play with other icon overrides defined e.g. in app or head, etc...

Basically:

1. we could define all kinds of complex fallbacks/overrides
2. we should comment in each one why or what it is testing
3. when the app has loaded, we run a last script at the end of page.js to analyse and test window.STATEMODULE to see if the state that was built by the execution of all fallbacks/overrides is the correct one

Of course, that testscript should have a hard coded hand crafted version of the final state we expect for all the instances and modules in our app and it could walk window.STATEMODULE to compare if our expectations are met.

So we first have to "run" all modules and fallbacks in our head and use the comments to build that hand crafted hard coded expectation in the test script.
The hard coded expectations in test script we compare against should also have comments to mention which module and which fallbacks are meant to be responsible for our expectation.

---

STATEMODULE

Cool that it has a .tree property to check if the constructed version is the correct one :-)

---

Looking forward for also fixing the "pre execution" you mention.