Is it possible to use files stored inside github repo?

[AuJezus]

Hello, I really like velite, it's a great project so please keep working on it :).

I would like to store my files on github repo, so I could use this as hosting for my .mdx files and each website that will consume these files will be able to pass it's own react components to render to html.

I know it's possible to get files from github using contentLayer, but I would like to use velite.

[zce]

Currently, Velite does not support remote content sources.

You can consider using git submodule with automated workflows to implement it.

[AuJezus]

Thank you, I will try to do that.

[iamnotbobby]

Currently, Velite does not support remote content sources.

You can consider using git submodule with automated workflows to implement it.

Will Velite supporting remote content sources be an expected feature in the later future?

[zce]

At present, we can only say that it is possible,

The current consideration is to implement it through a custom loader.

Although the loader architecture is not designed to support this.

maybe future:

import { defineLoader } from 'velite'

export default defineLoader({
  // name: 'json',
  test: /^http\:\/\//,
  load: pattern => {
    // fetch data from remote
    return new VFile({ ... })
  }
})

[aifirstd3v]

At present, we can only say that it is possible,

The current consideration is to implement it through a custom loader.

Although the loader architecture is not designed to support this.

maybe future:

import { defineLoader } from 'velite'

export default defineLoader({
  // name: 'json',
  test: /^http\:\/\//,
  load: pattern => {
    // fetch data from remote
    return new VFile({ ... })
  }
})

It would be so awesome if possible.
I've just asked for a coding example for remote content sources using chatgpt o1.

Velite does not officially support remote content sources, but you can approximate that functionality by creating a custom loader. The idea is to treat a remote URL as if it were a local file, then make an HTTP request within the loader to retrieve and parse the data. Although the loader architecture is not designed exclusively for network operations, the example below illustrates a possible approach.

• Define the custom loader by specifying a test pattern that matches http:// or https:// URLs, and use fetch (or any other HTTP request method) to retrieve remote data.
• Parse the response into a suitable shape for Velite (for example, a frontmatter-like object with data and content fields).
• Return this object so that Velite can integrate the data into its build pipeline.

Example code:

import { defineConfig, defineLoader } from 'velite'
import { VFile } from 'vfile'

const remoteJsonLoader = defineLoader({
  // This pattern will capture URLs that begin with http:// or https://
  test: /^https?:\/\//,
  load: async vfile => {
    // The URL is stored in vfile.path
    const url = vfile.path

    try {
      const response = await fetch(url)
      if (!response.ok) {
        throw new Error(`Failed to fetch data from ${url} (status: ${response.status})`)
      }
      const text = await response.text()

      let data
      try {
        data = JSON.parse(text)
      } catch (err) {
        throw new Error(`Invalid JSON structure from ${url}`)
      }

      // Return an object that Velite recognizes.
      // Typically, { data, content } or a similar structure
      return {
        data, // This can be treated as frontmatter data
        content: '' // Remote JSON might have no separate "content" body
      }
    } catch (err) {
      throw new Error(`Remote JSON loader error: ${err.message}`)
    }
  }
})

export default defineConfig({
  collections: {
    posts: {
      // The pattern array may include both local and remote references
      pattern: [
        'posts/**/*.md',
        'https://example.com/posts.json'
      ],
      // Additional schema, transform, or collection settings can be defined here
    }
  },
  loaders: [
    remoteJsonLoader
  ]
})

Usage details:

• Place the custom loader code in your Velite configuration file (for instance, velite.config.js).
• In the collections object, add one or more remote URLs to the pattern array.
• Run the usual Velite commands such as velite build; Velite will detect that certain paths match the custom loader’s test condition, call the loader’s load function, and incorporate the returned data into its build pipeline.
• Since network requests can be slow or error-prone, ensure that you handle potential failures (e.g., HTTP status codes and invalid JSON) gracefully. This prevents unexpected interruptions during the build.

---

To enable Velite to read remote JSON as if it were local MDX content, you need to structure the remote JSON in a format that Velite can process. Velite typically handles content as objects with { data, content } format, where data acts like frontmatter metadata and content contains Markdown or MDX text. Below is an explanation and implementation example.

---

Example: Remote JSON Format

Assume the remote JSON is stored at https://example.com/posts.json with the following structure:

[
  {
    "data": {
      "title": "My First Remote Post",
      "date": "2024-06-01",
      "slug": "my-first-remote-post"
    },
    "content": "# Hello from Remote\n\nThis is MDX content stored remotely.\n"
  },
  {
    "data": {
      "title": "My Second Remote Post",
      "date": "2024-06-02",
      "slug": "my-second-remote-post"
    },
    "content": "# Another Remote Post\n\nHere is another example of remote MDX content.\n"
  }
]

• data field: Contains metadata for the content (e.g., title, date, slug) that Velite can interpret as frontmatter.
• content field: Contains MDX or Markdown content.

The Custom Loader will treat each object in this JSON array as an individual document.

---

Custom Loader Implementation

The following example shows how to create a Custom Loader to fetch remote JSON and integrate it into Velite's pipeline.

import { defineConfig, defineLoader } from 'velite'
import { VFile } from 'vfile'

const remoteJsonLoader = defineLoader({
  // Match URLs starting with http:// or https://
  test: /^https?:\/\//,
  load: async vfile => {
    const url = vfile.path
    try {
      const response = await fetch(url)
      if (!response.ok) {
        throw new Error(`Failed to fetch data from ${url} (status: ${response.status})`)
      }
      const text = await response.text()

      let jsonData
      try {
        // Assume the JSON structure is an array of objects with { data, content } fields
        jsonData = JSON.parse(text)
      } catch (err) {
        throw new Error(`Invalid JSON structure from ${url}`)
      }

      // If Velite supports processing multiple documents, return an array of vfiles
      const vfiles = jsonData.map((item, idx) => ({
        data: item.data,
        content: item.content,
        path: `${url}?doc=${idx}` // Assign a unique path for each document
      }))
      return vfiles
    } catch (err) {
      throw new Error(`Remote JSON loader error: ${err.message}`)
    }
  }
})

export default defineConfig({
  collections: {
    posts: {
      name: 'Post',
      pattern: [
        'posts/**/*.md', // Local files
        'https://example.com/posts.json' // Remote JSON source
      ],
      schema: ({ s }) =>
        s.object({
          data: s.any(), 
          content: s.mdx() // Interpret content as MDX
        })
    }
  },
  loaders: [remoteJsonLoader]
})

---

Rendering in a Next.js Project

After running velite build, the collection output is available in the .velite directory. In a Next.js project, you can render the remote content just like local MDX files.

import { posts } from '@/.velite'

export default function Page() {
  return (
    <main>
      {Array.isArray(posts) &&
        posts.map((post, i) => (
          <article key={i}>
            <h2>{post.data.title}</h2>
            {/* For rendering MDX content, use a component that dynamically executes the MDX code */}
          </article>
        ))}
    </main>
  )
}

---

Key Considerations

1. Remote JSON Structure:

   • The JSON should have { data, content } format for each document.
   • The content field must contain valid MDX or Markdown syntax.

2. Handling Multiple Documents:

   • If the remote JSON contains multiple entries, return them as an array of vfile objects in the loader.

3. MDX Rendering:

   • Use a Next.js-compatible MDX renderer (e.g., a function that evaluates the MDX code in content).

4. Build Speed:

   • Fetching data from remote sources may slow down the build process depending on network latency.

5. Error Handling:

   • Properly handle HTTP errors, invalid JSON structures, and missing fields to prevent build failures.

By following this approach, remote JSON data can be seamlessly integrated into Velite, treated like local MDX files, and rendered as part of your Next.js blog or application.

[nrjdalal]

I made gitpick, a CLI tool to sync directories from GitHub repositories and more.

Run npx gitpick <github-url> [optional-target] to try it out. Let's go!

---

Say your files are located at a valid GitHub URL with a directory:

https://github.com/zce/velite/tree/main/docs

You can sync them to a target directory in your project using:

npx gitpick https://github.com/zce/velite/tree/main/docs awesome-docs

If no target directory is given, it will use the last segment of the URL, e.g., in this case, docs.

---

If you need to watch for changes, just add -w or -w [time] (e.g., 1s, 1m, 1h which is time in a human-readable format, default is 1m):

gitpick https://github.com/zce/velite/tree/main/docs -w

---

Additionally, you can use this tool to clone any file or directory from any repository. For example:

gitpick https://github.com/nrjdalal/awesome-templates/tree/main/next.js-apps/next.js-pro

The above command will clone the next.js-pro template from that particular repository.

---

Run gitpick -h to get help with usage.

With gitpick, you can clone precisely what you need.

🚀 More awesome tools at https://github.com/nrjdalal

-------------------------------------
  gitpick <url> [target] [options]
-------------------------------------

Hint: Target is optional, and follows default git clone behavior.

Arguments:
  url                GitHub URL with path to file/folder
  target             Directory to clone into (optional)

Options:
  -b, --branch       Branch to clone
  -o, --overwrite    Skip overwrite prompt
  -w, --watch [time] Watch the repository and sync every [time]
                     e.g. 1h, 30m, 15s) default: 1m
  -h, --help         display help for command
  -v, --version      display the version number

Examples:
  $ gitpick <url>
  $ gitpick <url> [target]
  $ gitpick <url> [target] -b [branch]
  $ gitpick <url> [target] -w [time]
  $ gitpick <url> [target] -b [branch] -w [time]

I would love feedback from you all. Let's go!

[nrjdalal]

Hello, I really like velite, it's a great project so please keep working on it :).

I would like to store my files on github repo, so I could use this as hosting for my .mdx files and each website that will consume these files will be able to pass it's own react components to render to html.

I know it's possible to get files from github using contentLayer, but I would like to use velite.

@AuJezus checkout gitpick's watch mode to do just that

• will add gitpick.config.js later today
• will see how to integrate with projects directly, currently you can use concurrently or run in diff tab

[iamnotbobby]

I made gitpick, a CLI tool to sync directories from GitHub repositories and more.

Run npx gitpick <github-url> [optional-target] to try it out. Let's go!

Say your files are located at a valid GitHub URL with a directory:

https://github.com/zce/velite/tree/main/docs

You can sync them to a target directory in your project using:

npx gitpick https://github.com/zce/velite/tree/main/docs awesome-docs

If no target directory is given, it will use the last segment of the URL, e.g., in this case, docs.

If you need to watch for changes, just add -w or -w [time] (e.g., 1s, 1m, 1h which is time in a human-readable format, default is 1m):

gitpick https://github.com/zce/velite/tree/main/docs -w

Additionally, you can use this tool to clone any file or directory from any repository. For example:

gitpick https://github.com/nrjdalal/awesome-templates/tree/main/next.js-apps/next.js-pro

The above command will clone the next.js-pro template from that particular repository.

Run gitpick -h to get help with usage.

With gitpick, you can clone precisely what you need.

🚀 More awesome tools at https://github.com/nrjdalal

-------------------------------------
  gitpick <url> [target] [options]
-------------------------------------

Hint: Target is optional, and follows default git clone behavior.

Arguments:
  url                GitHub URL with path to file/folder
  target             Directory to clone into (optional)

Options:
  -b, --branch       Branch to clone
  -o, --overwrite    Skip overwrite prompt
  -w, --watch [time] Watch the repository and sync every [time]
                     (e.g. 1h, 30m, 15s) default: 1m
  -h, --help         display help for command
  -v, --version      display the version number

Examples:
  $ gitpick
  $ gitpick [target]
  $ gitpick [target] -b
  $ gitpick [target] -w [time]

I would love feedback from you all. Let's go!

This is awesome!

[nrjdalal]

@bobbytxt do try and let me know if it's working for you

[nrjdalal]

You can put gitpick in your velite.config.js to continuously sync with GitHub:

import { defineConfig, s } from 'velite'
import { spawn } from 'node:child_process'

spawn('npx', ['gitpick', 'zce/velite', 'content/github', '-w'])
  .stdout.on('data', data => console.log(`${data}`))

/*
this will sync zce/velite repository, to ./content/github
you can sync directories with gitpick too, check repo out for usage
*/

export default defineConfig({
  // ... velite.config.js
})

When you need to sync once or during build, just switch -w with -o and use exec. This will act as a remote source updater for GitHub.

If you need any help with that, feel free to reach out to me at X.

[aifirstd3v]

You can put gitpick in your velite.config.js to continuously sync with GitHub:

import { defineConfig, s } from 'velite'
import { spawn } from 'node:child_process'

spawn('npx', ['gitpick', 'zce/velite', 'content/github', '-w'])
.stdout.on('data', data => console.log(${data}))

/*
this will sync zce/velite repository, to ./content/github
you can sync directories with gitpick too, check repo out for usage
*/

export default defineConfig({
// ... velite.config.js
})
When you need to sync once or during build, just switch -w with -o and use exec. This will act as a remote source updater for GitHub.

If you need any help with that, feel free to reach out to me at X.

compatible with edge runtime?

[nrjdalal]

You can put gitpick in your velite.config.js to continuously sync with GitHub:
import { defineConfig, s } from 'velite'
import { spawn } from 'node:child_process'
spawn('npx', ['gitpick', 'zce/velite', 'content/github', '-w'])
.stdout.on('data', data => console.log(${data}))
/*
this will sync zce/velite repository, to ./content/github
you can sync directories with gitpick too, check repo out for usage
*/
export default defineConfig({
// ... velite.config.js
})
When you need to sync once or during build, just switch -w with -o and use exec. This will act as a remote source updater for GitHub.
If you need any help with that, feel free to reach out to me at X.

compatible with edge runtime?

what exactly does that mean? care to explain a little more