Skip to content

Commit

Permalink
doc and cleanup
Browse files Browse the repository at this point in the history
  • Loading branch information
fbeauchamp committed Feb 11, 2025
1 parent 2d0c8e4 commit 1c29469
Show file tree
Hide file tree
Showing 8 changed files with 153 additions and 49 deletions.
3 changes: 0 additions & 3 deletions @vates/fatfs/.gitignore

This file was deleted.

12 changes: 7 additions & 5 deletions @vates/fatfs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@
"postversion": "npm publish --access public"
},
"repository": {
"directory": "@vates/fatfs",
"type": "git",
"url": "https://github.com/vatesfr/xen-orchestra"
"url": "https://github.com/vatesfr/xen-orchestra.git"
},
"keywords": [
"filesystem",
Expand All @@ -20,13 +21,14 @@
],
"author": "Vates SAS",
"license": "Apache-2.0",
"bugs": {
"url": "https://github.com/vatesfr/xen-orchestra/issues"
},
"homepage": "https://github.com/vatesfr/xen-orchestra",
"bugs": "https://github.com/vatesfr/xen-orchestra/issues",
"homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@vates/fatfs",
"dependencies": {
"fifolock": "^1.0.0",
"struct-fu": "^1.2.1",
"xok": "^1.0.0"
},
"engines": {
"node": ">=8.10"
}
}
59 changes: 59 additions & 0 deletions @vates/generator-toolbox/.USAGE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
A toolbox to ease the use of generator

### Timeout

wrap a source async generator to have it throw an error when timeout is reached
timeout is a positive number in milliseconds

```js
import { Timeout } from '@vates/generator-toolbox'

const wrappedGenerator = new Timeout(sourceGenerator, timeout)
```

### Throttle

wrap a source async generator to have it respect a max speed ( in bytes per seconds).
speed is either a strictly positive number or a function returning a strictly positive number. A speed change will be used for the next emitted packet.

The source generator must yield object with a length property.

Optimized for small yields regarding to the speed, since it won't split incoming packet.

If the generator reached the max speed it will be paused, limiting memory consumption.

```js
import { Throttle } from '@vates/generator-toolbox'

const wrappedGenerator = new Throttle(sourceGenerator, speed)
```

### Synchronized

Fork a generator. The rules ares:

- if the source returns, all the forks returns
- if the forks errors, all the forks errors with the same error
- if a the fork return , it is stopped, but the generator continue with the other
- if a the fork error , it is stopped, but the generator continue with the other
- if all the fork return , the source is stopped
- if al the fork error , the source is errored with the last error
- the source start producing a packet when the fastest forked ask for it
- the source forks get the packet only when all the forks asked for it, no buffer stores in memory

```ts
import { Synchronized } from '@vates/generator-toolbox'

async function consume(
generator: AsyncGenerator {
for await (const val of generator) {
console.log({val})
}
}
const forker = new Synchronized(generator)
const first = forker.fork('first')
const second = forker.fork('second')
await Promise.all([consume(first), consume(second)])
```

Note: you can stop early a generator by calling `generator.return()`, and you can stop in in error by calling `generator.throw(error)`
35 changes: 0 additions & 35 deletions @vates/generator-toolbox/.npmignore

This file was deleted.

1 change: 1 addition & 0 deletions @vates/generator-toolbox/.npmignore
80 changes: 80 additions & 0 deletions @vates/generator-toolbox/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
<!-- DO NOT EDIT MANUALLY, THIS FILE HAS BEEN GENERATED -->

# @vates/generator-toolbox

## Usage

A toolbox to ease the use of generator

### Timeout

wrap a source async generator to have it throw an error when timeout is reached
timeout is a positive number in milliseconds
```js
import { Timeout } from '@vates/generator-toolbox'

const wrappedGenerator = new Timeout(sourceGenerator, timeout)

```

### Throttle

wrap a source async generator to have it respect a max speed ( in bytes per seconds).
speed is either a strictly positive number or a function returning a strictly positive number. A speed change will be used for the next emitted packet.

The source generator must yield object with a length property.

Optimized for small yields regarding to the speed, since it won't split incoming packet.

If the generator reached the max speed it will be paused, limiting memory consumption.

```js
import { Throttle } from '@vates/generator-toolbox'

const wrappedGenerator = new Throttle(sourceGenerator, speed)

```

### Synchronized

Fork a generator. The rules ares:

* if the source returns, all the forks returns
* if the forks errors, all the forks errors with the same error
* if a the fork return , it is stopped, but the generator continue with the other
* if a the fork error , it is stopped, but the generator continue with the other
* if all the fork return , the source is stopped
* if al the fork error , the source is errored with the last error
* the source start producing a packet when the fastest forked ask for it
* the source forks get the packet only when all the forks asked for it, no buffer stores in memory

```ts
import { Synchronized } from '@vates/generator-toolbox'

async function consume(
generator: AsyncGenerator {
for await (const val of generator) {
console.log({val})
}
}
const forker = new Synchronized(generator)
const first = forker.fork('first')
const second = forker.fork('second')
await Promise.all([consume(first), consume(second)])
```
Note: you can stop early a generator by calling `generator.return()`, and you can stop in in error by calling `generator.throw(error)`

## Contributions

Contributions are _very_ welcomed, either on the documentation or on
the code.

You may:

- report any [issue](https://github.com/vatesfr/xen-orchestra/issues)
you've encountered;
- fork and create a pull request.

## License

[MIT](https://spdx.org/licenses/MIT) © [Vates SAS](https://vates.fr)
2 changes: 1 addition & 1 deletion @vates/generator-toolbox/src/throttle.mts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import assert from 'node:assert'
* The source generator must yield object with a length property
* Changing the speed will only be takin into account fot the next packets asked
*/
export class GeneratorThrottler {
export class Throttle {
#previousSlot = 0
#bytesPerSecond: number | (() => number)
get speed(): number {
Expand Down
8 changes: 4 additions & 4 deletions @vates/generator-toolbox/src/throttle.test.mts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { GeneratorThrottler } from './throttle.mjs'
import { Throttle } from './throttle.mjs'
import { suite, test } from 'node:test'
import assert from 'node:assert'

Expand All @@ -17,7 +17,7 @@ async function consumes(generator: AsyncGenerator) {
suite('it throttle one', () => {
test('base test', async () => {
// 10MB/s
const throttler = new GeneratorThrottler(10 * 1024 * 1024)
const throttler = new Throttle(10 * 1024 * 1024)
const throttled = throttler.createThrottledGenerator(makeGenerator(1024 * 1024, 20))

const start = Date.now()
Expand All @@ -27,7 +27,7 @@ suite('it throttle one', () => {
})
test('variable speed test', async () => {
// 10MB/s
const throttler = new GeneratorThrottler(() => 10 * 1024 * 1024)
const throttler = new Throttle(() => 10 * 1024 * 1024)
const throttled = throttler.createThrottledGenerator(makeGenerator(1024 * 1024, 20))

const start = Date.now()
Expand All @@ -37,7 +37,7 @@ suite('it throttle one', () => {
})
test('multiple generator test', async () => {
// 10MB/s
const throttler = new GeneratorThrottler(10 * 1024 * 1024)
const throttler = new Throttle(10 * 1024 * 1024)
const first = throttler.createThrottledGenerator(makeGenerator(1024 * 1024, 10))
const second = throttler.createThrottledGenerator(makeGenerator(1024 * 1024, 10))
const start = Date.now()
Expand Down
2 changes: 1 addition & 1 deletion packages/vhd-lib/Vhd/VhdAbstract.integ.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ const streamToBuffer = stream => {

let tempDir

describe('VhdAbstract', { concurrency: 1 }, async () => {
describe('VhdAbstract', async () => {
beforeEach(async () => {
tempDir = await pFromCallback(cb => tmp.dir(cb))
})
Expand Down

0 comments on commit 1c29469

Please sign in to comment.