324 GitHub discussions with graph ql api

[crepesAlot]

No description provided.

[carlosparadis]

@crepesAlot You mentioned a few times something was not working on this PR but I don't remember ever going in detail with you. Do you want to share what is it that you found was not working to see if I can help?

[crepesAlot]

@carlosparadis Yes, I'm having difficulties getting the refresher to work properly.
The main problem is this line:

  # Read the JSON file
  json_data <- fromJSON(file.path(save_folder_path, latest_discussion), simplifyVector = FALSE)

The error I get is

Error: lexical error: invalid char in json text.
                                       NA
                     (right here) ------^

The problem seems to stem from the way I check the folder the downloaded JSON files should be in.

  # If there are no json files, download all discussions
  if (length(contents) == 0 ) {
    # Run the regular downloader
    discussions <- github_api_discussions(token, owner, repo, save_folder_path)
    return (discussions)
  }

However, if I delete the JSON files in the save_folder_path (thus making it empty) it does not run the regular downloader.
running:

contents <- list.files(save_folder_path, all.files = TRUE)
print(contents)
print(length(contents)==0)
[1] "."         ".."        ".DS_Store"
[1] FALSE

I believe I need to somehow make the function explicitly check for JSON files, rather than if the folder is empty.
My attempts at this were:

json_contents <- list.files(save_folder_path, pattern= "\\.json$", full.names = TRUE, recursive= TRUE)

However this also causes some problems as it adds an extra slash to the folder_path like so:

"../../rawdata/deep-live-cam/github/hacksider_deep-live-cam/discussion//hacksider_Deep-Live-Cam_1714152212_1732289412.json"

I know this isn't caused by an incorrect save_folder_path because printing the save_folder_path returs

"../../rawdata/deep-live-cam/github/hacksider_deep-live-cam/discussion/"

Explicitly checking for the JSON files solves the issue of the refresh not running the regular downloader if the folder is empty, but throws the same error if the folder is not empty, (thus necessitating it to check if the downloaded files are up to date)

[carlosparadis]

@crepesAlot have you checked how the JIRA downloader handles this? It also downloads json files. You may even be able to reuse the entire code logic from there.

[crepesAlot]

@carlosparadis I found out the solution was simple all along. I simply needed to remove the all.files=TRUE from the list.files.
Unfortunately, I found some new errors that I need to fix as well.
When I delete one of the JSON files (to simulate the folder being out-of-date) it doesn't properly update and download the most recent discussions.
I tried to follow the jira and github refresh functions as closely as I could, but because the graphql functions off of query, the current problem with the code is with the way I am forming said query, which is something I cannot copy from the other refresh functions.

[carlosparadis]

Just so I am clear, when you say it does not update: Are you trying to edit the existing file? You don't need to edit the existing file, it is ok the new changes are added as a new file, provided the timestamp of the file is correct.

The files do not need to be precisely per month. I do not believe JIRA nor GitHub obey that either. Dao's is per month only because mbox files are downloaded per month only.

[crepesAlot]

@carlosparadis No, I am testing with a repository that the downloader creates 2 new JSON files, then deleting 1 of the JSON files with the more recent discussions. I then try to run the refresher to see if it will pick up that there are more recent discussion than the remaining JSON file in the folder.
But the refresh function doesn't detect this, and returns that there are no new discussions to download.

I strongly believe this is because I structured the query in the refresh incorrectly.
I am having some difficulties with trying to fix this due to some confusion with how the GraphQL retrieves the discussions from Github.

discussions(
  after: String,
  before: String,
  first: Int,
  last: Int,
  categoryId: ID = null,
  answered: Boolean = null,
  orderBy: DiscussionOrder = {field: UPDATED_AT, direction: DESC}
) : Discussion

I had thought that the query would work by providing it the date at the "after" field, but that doesn't seem to work.
The queries seem to rely on a cursor, a type of string retrieved from a different query.

[carlosparadis]

@crepesAlot You should not have delete file logic! This will overly complicate things. Just have it so the new file starts of where the last file began, which information can be obtained from the file name. Trying to edit an existing JSON to add more data is the road to madness!

I'm still trying to understand the cursor part. Are you saying we can't query discussion starting from a created at date? So long that is possible, then you can just create the new file. Please remove the delete file logic, it will be hard to maintain with file deletion, which is why GitHub, JIRA and Bugzilla just add new files.

[crepesAlot]

@carlosparadis The function doesn't delete or modify any JSON files. The downloader acts exactly as you say, it downloads 1 page, saves it as a JSON, then goes to page 2 and saves that as a new, separate JSON file.

The act of manually deleting a file (me going into my Finder and deleting it) was my attempt to forcibly try to see if it could check if the dates of the JSON file were not the latest discussion, then download the discussions from where the remaining JSON left off. Which is what didn't work.
I wasn't sure of another way to test it without posting random things in this repo's discussion, but I was able to find that the query in the refresh function doesn't work the way I thought it did regardless.

We cannot directly form a query using a createdAt date.
I am currently trying to see if there's still a way to implement a refresher function by the way the query orders the discussions and saves the cursor.
From my understanding the cursor acts as a sort of page number for the query. (Since the number of discussions a query can get per page can vary from 1 to 100). But it currently is up in air as to whether I can get it working or not.