Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: refactor to use tb locally #21952

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

javisantana
Copy link

A refactor of the tinybird project to:

  1. get rid of materialized views and use JSON field. It'll help with project maintenance, you only need to version the pipes and that'll work no matter how often open source users update

  2. use the new cli that allow to build and test the project locally (instead of pushing to a branch)

TODO: I still need to figure out how the tinybird project test are run

$ curl -LsSf https://api.tinybird.co/static/install.sh | sh
$ tb local start
$ ./scripts/branch_and_test.sh 
» Building project...

** Running 'analytics_events' 
** 'analytics_events' created
** Running 'analytics_sources' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/analytics_sources.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'analytics_sources' created
** Running 'analytics_sessions' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/analytics_sessions.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'analytics_sessions' created
** Running 'analytics_pages' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/analytics_pages.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'analytics_pages' created
** Running 'trend__v0' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/trend__v0.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'trend__v0' created
** Running 'top_sources__v0' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/top_sources__v0.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'top_sources__v0' created
** Running 'top_pages__v0' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/top_pages__v0.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'top_pages__v0' created
** Running 'top_browsers__v0' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/top_browsers__v0.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'top_browsers__v0' created
** Running 'top_devices__v0' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/top_devices__v0.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'top_devices__v0' created
** Running 'kpis__v0' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/kpis__v0.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'kpis__v0' created
** Running 'top_locations__v0' 
** => Test endpoint with:
** $ curl http://localhost:80/v0/pipes/top_locations__v0.json?token=p.eyJ1IjogIjUyN2ZkM2FiLTk2ZTEtNGQzMi04NTQ4LThiMTgwMDY2YjkwZCIsICJpZCI6ICJkMWJhZDE5OS1jYWY1LTRlOWMtOGViOS0zMjI4NmY0NWNlNDMiLCAiaG9zdCI6ICJsb2NhbCJ9.aXCWWGh_nC74cYDwJpAaJ5NtdF1wGozWzUE7eohirQw
** 'top_locations__v0' created

✓ Build completed in 3.6s
++ directory=datasources/fixtures
++ extensions=("csv" "ndjson")
+++ realpath datasources/fixtures
++ absolute_directory=/Users/javisantana/tmp/Ghost/ghost/tinybird/datasources/fixtures
++ for extension in '"${extensions[@]}"'
+++ find /Users/javisantana/tmp/Ghost/ghost/tinybird/datasources/fixtures -type f -name '*.csv'
++ file_list=
++ for extension in '"${extensions[@]}"'
+++ find /Users/javisantana/tmp/Ghost/ghost/tinybird/datasources/fixtures -type f -name '*.ndjson'
++ file_list=/Users/javisantana/tmp/Ghost/ghost/tinybird/datasources/fixtures/analytics_events.ndjson
++ for file_path in '$file_list'
+++ basename /Users/javisantana/tmp/Ghost/ghost/tinybird/datasources/fixtures/analytics_events.ndjson
++ file_name=analytics_events.ndjson
++ file_name_without_extension=analytics_events
++ command='tb --local datasource append analytics_events datasources/fixtures/analytics_events.ndjson'
++ echo tb --local datasource append analytics_events datasources/fixtures/analytics_events.ndjson
tb --local datasource append analytics_events datasources/fixtures/analytics_events.ndjson
++ tb --local datasource append analytics_events datasources/fixtures/analytics_events.ndjson

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant