Reputation Mining Visualizer, Part 1 #582

kronosapiens · 2019-04-03T14:50:36Z

Closes #550 (but not the end of road)

Changes Summary

This PR includes two .html files which implement self-contained visualizers for reputation state (no transpiling / packing required!), served by the reputation miner (as well as an index.html for the mining client)

The first, repCycle.html, supports viewing of a ReputationMiningCycle contract, specifically the log entries and dispute state.

The second, repTree.html supports viewing of the reputation tree accessed via a mining client.

Neither is complete yet, lots of room to grow, but both do something so let's get this party started!

Setup Instructions

To visualize data from an existing network, simply spin up a mining client which points to the network in question. To generate test data on a local ganache network, run yarn run viz:bootstrap and then paste the resulting command into a new terminal.

Once the miner is running, navigate to http://localhost:3000/.

Alternatively, spin up a mining client and point it to the Görli testnet to view live testnet data.

Screenshots

Home Page

Rep Cycle Explorer

Rep Tree Visualizer

elenadimitrova · 2019-05-13T11:09:31Z

packages/reputation-miner/viz/repCycle.html

+    <div id="app">
+      <h2>Welome to RepMinCycViz!</h2>
+      <h3>Mining Cycle Address:</h3>
+      <input placeholder="Mining Cycle Address" v-model="cycleAddress">


Will this work across mining cycles? i.e. when the cycle completes and the cycleAddress changes?

Currently you have to manually update the mining cycle address. Over the weekend I was thinking of ways to have the visualizer automatically fetch the address -- doing the fetch on load should be no problem, but automatically refreshing when the cycle updates would be a bit more involved. At minimum refreshing the visualizer should give you the current mining cycle.

In #596 we had a similar requirement and ended up listening for the ReputationMiningCycleComplete event. These 2 PRs are very close in that they improve the quality of life of reputation miners so perhaps we can rebase this on top of #596 and hook into that event handler as well?

Mm I'll look into it. It may not be necessary though, since unlike the miner which is meant to be running automatically in the background, users will generally be interacting with the visualizer manually, so a manual refresh (especially given that mining cycles are an hour+) doesn't seem too bad. Although I could imagine someone potentially wanting to leave the visualizer running on a screen for a longer period of time, in which case the automatic update would be useful.

Although I'd still say that we should merge this as-is and we can add the auto-update functionality in a later PR.

elenadimitrova · 2019-05-13T11:12:18Z

packages/reputation-miner/viz/setup.js

@@ -0,0 +1,99 @@
+/* globals artifacts */
+const { WAD, MINING_CYCLE_DURATION } = require("../../../helpers/constants");


The setup here is in fact only hydrating some test data in the current cycle. I don't think it's necessary except for testing. In practice the visualiser should probably be a part of the ReputationMiningClient and started in its constructor or initialise call.

You're right, it's only for testing. I can move it elsewhere to make that more clear.

Maybe it fits in the test-data-generator.js? Or we don't need it at all? For development purposes we can also hook into the now live Goerli deployment where there are already some test log entries to process.

I'd say leave it for now, since having more precise control over the test data will be useful for at least a while longer. I'd say leave it in scripts since truffle scripts run in a different environment than the test code.

elenadimitrova

Tested against the current Goerli reputation state and looks lovely:

Really good work on this @kronosapiens 👏

elenadimitrova · 2019-05-16T07:38:17Z

packages/reputation-miner/viz/repCycle.html

+          parseLogEntry: function (logEntry) {
+            return {
+              user: logEntry.user.slice(0, 8),
+              amount: ethers.utils.formatEther(logEntry.amount._hex),


This formatEther gives me back for a value of 10 -> "amount": "0.00000000000000001"` which I find counter intuitive. Also we're not talking about Ether value here but reputation or token amounts which might not be 18 decimals. My preference for now would be to show the actual value but open to other suggestions.

elenadimitrova · 2019-05-16T07:38:51Z

packages/reputation-miner/viz/repCycle.html

+          },
+          parseLogEntry: function (logEntry) {
+            return {
+              user: logEntry.user.slice(0, 8),


Please please don't slice the user! Seeing "user": "0xb77D57" is really unhelpful

Fair, I did that at the beginning so the entries would easily render on one line but it's not a great production feature.

elenadimitrova · 2019-05-16T07:39:17Z

packages/reputation-miner/viz/repCycle.html

+              user: logEntry.user.slice(0, 8),
+              amount: ethers.utils.formatEther(logEntry.amount._hex),
+              skillId: parseInt(logEntry.skillId._hex),
+              colony: logEntry.colony.slice(0, 8),


What's with the slicing of addresses?

Same as above, I wanted things to render once-per-line for development.

elenadimitrova · 2019-05-16T07:42:48Z

packages/reputation-miner/viz/repCycle.html

+        <option>Inactive Cycle</option>
+      </select>
+
+      <p>Cycle Address: {{ cycleAddress }}  </p>


Can we add the open timestamp and maybe remaining time to close here? I was needing this information this morning to determine where my updates from 10am have gone.

elenadimitrova · 2019-05-16T07:45:12Z

packages/reputation-miner/viz/repCycle.html

+          miningCycle: {},
+          numEntries: 0,
+          logEntries: [],
+          numSubmittedHashes: 0,


Can we rename this to numUniqueSubmittedHashes everywhere including the way it displays Num Submitted Hashes -> Num Unique Submitted Hashes. We renamed this in the last few days to differentiate better between the number of submissions of a particular hash (up to 12) and the number of unique hashes.

elenadimitrova · 2019-05-16T07:49:22Z

packages/reputation-miner/viz/repCycle.html

+    <div id="app">
+      <h2>Welome to RepMinCycViz!</h2>
+      <select v-model="selected">
+        <option>Active Cycle</option>


There is a problem with the initial load, although the "Active cycle" appears selected it is not loaded in the visualiser. I don't particularly mind refreshing the dropdown but if we can auto-load this the first time it'll be better.

I couldn't immediately think of a clean way to do that in Vue, but I can think harder.

elenadimitrova · 2019-05-16T08:45:35Z

packages/reputation-miner/viz/repCycle.html

+<html lang="en">
+  <head>
+    <meta charset="utf-8">
+    <title>Colony Mining Cycle Visualizer</title>


Can we have a non-comic shrift font for this please :)

a l r i g h t

elenadimitrova · 2019-05-16T08:56:29Z

packages/reputation-miner/viz/repTree.html

@@ -0,0 +1,320 @@
+<!-- Forked from: https://bl.ocks.org/d3noob/43a860bc0024792f8803bba8ca0d5ecd -->


Can we display the reputation amounts as the full value

elenadimitrova · 2019-05-16T08:57:52Z

packages/reputation-miner/viz/repCycle.html

+    <!-- Page HTML -->
+    <div id="app">
+      <h2>Welome to RepMinCycViz!</h2>
+      <select v-model="selected">


Can we add the current reputation root hash here please

Is that from the ColonyNetwork contract or the ReputationMiningCycle? If the former, it would require bringing the ColonyNetwork ABI into the visualizer which I'd like to avoid if possible since it's more data and complexity (although doable if need be).

It would need to come from ColonyNetwork

What's the benefit of being able to see the root hash itself?

The most obvious benefit I can see other than just being nice to see is that you would then have all the information needed to query the oracle to get the merkle proofs (maybe the reputation values could even link to the relevant page on the local oracle?)

elenadimitrova · 2019-05-16T08:58:16Z

packages/reputation-miner/viz/repTree.html

@@ -0,0 +1,320 @@
+<!-- Forked from: https://bl.ocks.org/d3noob/43a860bc0024792f8803bba8ca0d5ecd -->
+<!DOCTYPE html>


Also 0 values aren't displayed atm and should be

Will investigate.

Ok should be good now.

kronosapiens · 2019-05-21T08:51:56Z

kronosapiens · 2019-05-22T07:47:10Z

Update: including root hash when rendering tree (currently the miner's current root hash).

kronosapiens · 2019-05-22T07:50:49Z

Some open todos:

Use Bootstrap to give the mining cycle page a proper UX treatment.
Extend the tree visualizer to be able to accept and render data for arbitrary root hashes.
Add support for dispute data.

elenadimitrova

Just a couple of remaining bits and we can merge

elenadimitrova · 2019-05-22T07:57:42Z

docs/_Docs_Releases.md

 ---

-![glider](/docs/img/glider_fullColor.svg)
+![glider](img/glider_fullColor.svg)


This is already part of #614

Ok, I'll remove.

elenadimitrova · 2019-05-22T07:58:59Z

package.json

    "posttest:contracts:watch": "npm run stop:blockchain:client",
-    "posttest:contracts:security": "sed -ie \"s/docker: false/docker: true/g\" ./truffle.js && rimraf ./truffle.jse"
+    "posttest:contracts:security": "sed -ie \"s/docker: false/docker: true/g\" ./truffle.js && rimraf ./truffle.jse",
+    "viz:bootstrap": "npm run start:blockchain:client & truffle migrate --reset --compile-all && truffle exec --network development scripts/viz-bootstrap.js"


Did we not decide the bootstrapping was unnecessary?

No, we still disagree. I'd like to leave it in for now, we can remove it once the visualizer is more mature.

elenadimitrova · 2019-05-22T08:01:18Z

packages/reputation-miner/viz/README.md

@@ -0,0 +1,15 @@
+# Welcome to the Colony Mining visualizers!


I'd argue we remove this readme and incorporate information on running the visualiser in the network docs for the miner. Otherwise this feels too separate from the client and I'd prefer to think of them as the UI and logic to the same thing.

Ok, I can move the readme.

elenadimitrova · 2019-05-22T08:16:55Z

Looks much better

kronosapiens added the in-progress label Apr 3, 2019

kronosapiens self-assigned this Apr 3, 2019

kronosapiens force-pushed the feature/cycle-viz branch from dbb712e to d261fee Compare April 4, 2019 07:58

kronosapiens force-pushed the feature/cycle-viz branch 5 times, most recently from d5d47ab to b96e894 Compare April 12, 2019 16:27

kronosapiens force-pushed the feature/cycle-viz branch 2 times, most recently from 9961f2e to 52ca639 Compare April 28, 2019 07:34

kronosapiens force-pushed the feature/cycle-viz branch 2 times, most recently from b1f855a to 1929c8f Compare May 8, 2019 21:16

kronosapiens changed the title ~~Reputation Mining Visualizer~~ Reputation Mining Visualizer, Part 1 May 8, 2019

kronosapiens added ready-for-review and removed in-progress labels May 8, 2019

kronosapiens force-pushed the feature/cycle-viz branch from ca4b929 to e3232a1 Compare May 9, 2019 12:25

elenadimitrova added under-review and removed ready-for-review labels May 10, 2019

kronosapiens force-pushed the feature/cycle-viz branch from 108d7aa to 094df4f Compare May 10, 2019 12:38

elenadimitrova reviewed May 13, 2019

View reviewed changes

kronosapiens force-pushed the feature/cycle-viz branch 2 times, most recently from e3db835 to bb5fa49 Compare May 13, 2019 17:00

elenadimitrova reviewed May 16, 2019

View reviewed changes

elenadimitrova mentioned this pull request May 20, 2019

Docs updates for glider-rc.1 -- "Get Started" with remix and truffle #614

Merged

kronosapiens force-pushed the feature/cycle-viz branch 2 times, most recently from 75da52e to 6707e0e Compare May 21, 2019 08:28

kronosapiens added 2 commits May 22, 2019 10:47

Create setup script to mock ReputationMiningCycle data

536ea45

Visualizer for reputation log entries

04e1369

kronosapiens added 15 commits May 22, 2019 10:47

Get reputation miner to mine mocked reputation state

4aab615

First pass at reputation tree visualizer

f1cdb15

Try pure D3 tree

402507b

Add support for patricia tree key chunking

1cb4b58

Replace Vue tree with D3 tree

549f26f

Allow reputation miner to serve visualizers

f1f1f14

Reorganize visualizer files into mining package

2d0bcaf

Clean up repTree

72b7dc5

Remove mining cycle constants file

450b56d

Always cite your sources

18ffc8f

Make constants in maliciousMiner match regular miner

6a90077

Add index.html to reputation mining client

7bffb65

Move bootstrap script into scripts/

3f03c46

Update repCycle to more easily switch between mining cycles

5573a58

Improve bootstrap script output

9e8083a

kronosapiens force-pushed the feature/cycle-viz branch from 7b4a736 to 2e0e6e0 Compare May 22, 2019 07:47

elenadimitrova reviewed May 22, 2019

View reviewed changes

kronosapiens added 2 commits May 22, 2019 11:19

Update repCycle visualizer to support Infura providers

02b1e07

Update in response to review feedback

85ef14a

kronosapiens force-pushed the feature/cycle-viz branch from 2e0e6e0 to f17b605 Compare May 22, 2019 08:26

kronosapiens added 2 commits May 22, 2019 12:34

Include root hash in tree visualizer

f195b87

Add visualizer documentation to docs

8552dd7

kronosapiens force-pushed the feature/cycle-viz branch from f17b605 to 8552dd7 Compare May 22, 2019 09:34

elenadimitrova approved these changes May 22, 2019

View reviewed changes

elenadimitrova removed the under-review label May 22, 2019

elenadimitrova merged commit b13846d into develop May 22, 2019

elenadimitrova deleted the feature/cycle-viz branch May 22, 2019 13:04

kronosapiens mentioned this pull request May 23, 2019

Miner Visualizer Improvements #620

Open

5 tasks

		@@ -0,0 +1,99 @@
		/* globals artifacts */
		const { WAD, MINING_CYCLE_DURATION } = require("../../../helpers/constants");

		@@ -0,0 +1,320 @@
		<!-- Forked from: https://bl.ocks.org/d3noob/43a860bc0024792f8803bba8ca0d5ecd -->

		@@ -0,0 +1,320 @@
		<!-- Forked from: https://bl.ocks.org/d3noob/43a860bc0024792f8803bba8ca0d5ecd -->
		<!DOCTYPE html>

Reputation Mining Visualizer, Part 1 #582

Reputation Mining Visualizer, Part 1 #582

Uh oh!

Conversation

kronosapiens commented Apr 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Summary

Setup Instructions

Screenshots

Home Page

Rep Cycle Explorer

Rep Tree Visualizer

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kronosapiens May 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elenadimitrova left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kronosapiens commented Apr 3, 2019 •

edited

Loading

kronosapiens May 14, 2019 •

edited

Loading

kronosapiens May 20, 2019 •

edited

Loading

kronosapiens commented May 21, 2019 •

edited

Loading