Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Hashing & Release Info Providers [WIP] #1228

Draft
wants to merge 126 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
6374325
feat: add plugin based release providers [WIP]
revam Oct 6, 2024
d7da1e7
feat: allow plugins to provide which hashes were actually used for th…
revam Jan 10, 2025
fabb40a
today's progress
revam Feb 2, 2025
8cd4804
today's progress
revam Feb 8, 2025
726d0b3
today's progress
revam Feb 10, 2025
b3fddcb
refactor: add to mylist when release info have been added
revam Feb 14, 2025
d401f92
feat: add video relocated event to video service
revam Feb 14, 2025
0f43611
today's progress
revam Feb 16, 2025
737f047
feat: add generic import reason for user data save events
revam Feb 16, 2025
f140431
refactor: fix up removing releases
revam Feb 16, 2025
48cfad1
add database migration
revam Feb 23, 2025
aa40216
fix: don't save an attempt if no providers are enabled
revam Feb 23, 2025
28b7071
revert: unteach the providers/jobs about the abstraction interfaces
revam Feb 23, 2025
18c4ad4
fix up anidb release provider that I've procrastinated on so far
revam Feb 23, 2025
d37376c
don't limit process file job on anidb if anidb is disabled as a provider
revam Feb 23, 2025
d6dae94
move init of video release service to a more fitting place
revam Feb 23, 2025
689861d
fix adding type exports in loader
revam Feb 23, 2025
78933d1
fix sqlite attempts table
revam Feb 23, 2025
eee66b4
add missing custom type in stored release info map
revam Feb 23, 2025
37c70d9
move from shoko models to anidb models in the api for easier access t…
revam Feb 23, 2025
24012b8
fix up pococache
revam Feb 23, 2025
1c9ddef
fix: also fire providers updated event after providers have been added
revam Feb 23, 2025
bda868e
fix: enable anidb provider by default and disable all other providers…
revam Feb 23, 2025
1cc549c
rename GetProviderByID → GetProviderByName
revam Feb 23, 2025
2f4926a
move GetCurrentReleaseForVideo
revam Feb 23, 2025
c6a3ac7
add simple memory cache for anidb provider
revam Feb 23, 2025
dd9288b
change symbol in anidb releases from - to +
revam Feb 23, 2025
6b12df9
fix up anidb release id in db migration
revam Feb 23, 2025
4c36060
add basic APIv3 controller [WIP]
revam Feb 23, 2025
180c4a0
bump abstraction version
revam Feb 23, 2025
f4bbb68
add missing command and move enum values to not interfere with existi…
revam Feb 23, 2025
6e1d860
fix percentage ranges
revam Feb 23, 2025
f0d609b
use `Cache.CreateIndex` instead of `new PocoIndex`
revam Feb 23, 2025
5ca6d06
update poco cache
revam Feb 23, 2025
5f2bc36
remove AniDB_File model
revam Feb 23, 2025
cc1b0bc
chore: clean up unused namespaces
revam Feb 23, 2025
606b330
adjust to changes on 'master'
revam Feb 23, 2025
9bbfe88
today's progress
revam Feb 23, 2025
8f69d93
change bodies
revam Feb 24, 2025
e62f177
remove anidb file xref models
revam Feb 24, 2025
727ebba
adjust to changes on 'master'
revam Feb 24, 2025
ecbeaee
more logging in the anidb release provider
revam Feb 24, 2025
98a6b6b
fix release exporter
revam Feb 24, 2025
9bffec5
fix: fix description for last updated at, and remove it from release …
revam Feb 24, 2025
9d5f429
fix: save file size given by provider to disc
revam Feb 24, 2025
c352a8e
feat: store current file size & known hashes when using the anidb rel…
revam Feb 24, 2025
c93df5e
move embedded cross reference
revam Feb 24, 2025
2bffb21
fix: better embedded hashes
revam Feb 24, 2025
0a808f9
feat: add parallel mode (disabled by default for now)
revam Feb 24, 2025
8f80d00
refactor: rename ProviderID → ProviderName
revam Feb 24, 2025
4dbe5f4
refactor: generate unique IDs for release providers
revam Feb 25, 2025
01b216b
fix: store last updated at from existing release in release info with…
revam Feb 25, 2025
3af9d36
refactor: replace ol' file matched/not-matched event with release sea…
revam Feb 25, 2025
11fe992
refactor: remove file renamed/moved events on video service in favour…
revam Feb 25, 2025
245f3d1
refactor: simplify event init
revam Feb 25, 2025
994eb13
feat: add methods to get provider info for a provider instance & type
revam Feb 25, 2025
66a9379
refactor: change methods to get provider info
revam Feb 26, 2025
a90c26e
refactor: "smarter" parallel release finding logic
revam Feb 26, 2025
dffd403
feat: expose parameter to only return enabled providers
revam Feb 26, 2025
4e50c61
today's progress
revam Feb 27, 2025
bc70415
chore: rename `SVR_VideoLocal` → `VideoLocal`
revam Feb 27, 2025
f94faf5
fix: fix release importer/exporter
revam Feb 27, 2025
3a80bb1
fix: fix updating index/priority of providers
revam Feb 27, 2025
f5280f5
fix: cache enabled status for release exporter
revam Feb 27, 2025
b6f43db
fix: add missing file size to releases
revam Feb 27, 2025
51ee428
feat: expose hashing/release-info settings in the REST API
revam Feb 27, 2025
113c938
chore: `ReleaseInfoWithProvider` → `ReleaseInfo` in anidb provider
revam Feb 27, 2025
6558a78
refactor: include `ReleaseExporter` plugin by default
revam Feb 28, 2025
e92db0c
refactor: use plugin priority + more
revam Feb 28, 2025
dcfe1b7
chore: rename `BasePlugin` to `CorePlugin`
revam Feb 28, 2025
c862ba8
chore: remove ol' debug line
revam Feb 28, 2025
fc86123
misc: update version of release exporter
revam Feb 28, 2025
b823a70
chore: move plugins to plugin directory & simplify how to build plugins
revam Feb 28, 2025
615f981
feat: add relocation+ plugin, not built by default (for now)
revam Mar 1, 2025
ade69de
fix: don't schedule ProcessFileJob if auto matching is disabled
revam Mar 1, 2025
8cfaefa
fix: save changes to hashing service parallel mode
revam Mar 1, 2025
a7b91c4
fix: `and not` → `or` in db migration
revam Mar 1, 2025
0c73a61
fix: don't `continue`, set value to `null`
revam Mar 1, 2025
69c05c5
fix: break circular dependency
revam Mar 1, 2025
6e227ab
fix: add missing available/default hash types per provider in APIv3
revam Mar 1, 2025
b8a6f31
chore: revert back to 200 byte buffer for outputting
revam Mar 1, 2025
db7b843
fix: fix list checks in discover job
revam Mar 1, 2025
d58cbbd
refactor: add `IConfigurationService`, `ConfigurationProvider<TConfig…
revam Mar 2, 2025
299996e
misc: bump abstraction version
revam Mar 2, 2025
af44923
misc: allow selecting plugins to build when debugging
revam Mar 3, 2025
2e21227
chore: don't trigger side-effects when creating language settings object
revam Mar 3, 2025
69c0115
fix: replace properties when creating objects from string
revam Mar 3, 2025
4a4d80f
chore: remove outdated hasher settings
revam Mar 3, 2025
2eddf77
misc: add display section metadata support in JSON schema
revam Mar 3, 2025
a7f29d2
today's progress
revam Mar 4, 2025
11f74fb
chore: remove unnecessary imports
revam Mar 4, 2025
fd13243
misc: remove `.AsQueryable()` on methods returning `IEnumerable<>`
revam Mar 5, 2025
61009e6
fix: proper uuid to guid conversion
revam Mar 5, 2025
1e7a54c
fix: fix error message for config validation exception
revam Mar 5, 2025
3d7999a
fix: schema title for configs
revam Mar 5, 2025
510e4bd
today's progress
revam Mar 10, 2025
fc01df0
refactor: rename hide save action attribute
revam Mar 10, 2025
ca0b0ee
fix: fix property path enumeration in perform action internals
revam Mar 10, 2025
f5af376
feat: add field to signal to the client to fetch a copy of the config
revam Mar 10, 2025
373fddb
fix: catch validation errors thrown by a performed action
revam Mar 10, 2025
60158fa
misc: update shokofin config with more fake connect validation logic
revam Mar 10, 2025
cdafd5b
misc: remove unnecessary attribute for hidden field
revam Mar 10, 2025
88311d5
misc: add more groups/fields to the shokofin example config
revam Mar 10, 2025
8c90568
refactor: rename `IImportFolder` to `IManagedFolder` + more
revam Mar 10, 2025
2881936
fix: remove videos with no files attached during startup validation
revam Mar 11, 2025
db02e42
fix: allow unknown properties to exist in json
revam Mar 11, 2025
83bf7bc
refactor: remove settings saved event from shoko event handler
revam Mar 11, 2025
06183ee
feat: expose release match attempts & schedule release finding onto q…
revam Mar 11, 2025
5990365
refactor: upgrade `IAniDBService` to be more useful
revam Mar 11, 2025
b0b1338
fix: attempt to re-fetch anidb anime title cache if it's been more th…
revam Mar 11, 2025
70a79a2
refactor: move hashing responsibility from the job to the service, ad…
revam Mar 12, 2025
24fd5cf
fix: only schedule auto-match job if auto matching is enabled
revam Mar 12, 2025
6849c4f
fix: change id of core config before enumerating configs and config d…
revam Mar 12, 2025
ae92d55
misc: update obsolete message
revam Mar 12, 2025
3660885
fix: run managed folder events in a new thread
revam Mar 12, 2025
a18355b
fix: set static service provider/container _before_ loading plugins
revam Mar 12, 2025
1de2db7
chore: remove empty line
revam Mar 12, 2025
935d5cb
chore: fix typo in exception message
revam Mar 12, 2025
c6b08e4
refactor: move responsibility of refreshing anidb anime from the job …
revam Mar 12, 2025
3924254
refactor: remove defunct job parameter
revam Mar 12, 2025
cabda0b
fix: fix circular dependency loop i created
revam Mar 12, 2025
fe0ec6c
chore: rename video hashing service impl. to not have 'abstract' in i…
revam Mar 12, 2025
15f4678
chore: rename video release service impl. to not have 'abstract' in i…
revam Mar 12, 2025
3087b93
chore: rename AniDB service impl. to not have 'abstract' in it's name
revam Mar 12, 2025
d6f24a3
chore: rename user data service impl. to not have 'abstract' in it's …
revam Mar 12, 2025
4315e30
chore: more the remaining abstraction services into a new namespace/f…
revam Mar 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 7 additions & 0 deletions .vscode/tasks.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
"net8.0",
"${workspaceFolder}/Shoko.CLI/Shoko.CLI.csproj",
"/property:GenerateFullPaths=true,Version=${input:version},InformationalVersion=\"channel=debug,commit=${input:commit},date=${input:date}\"",
"/p:PluginNames=\"${input:pluginNames}\"",
"/consoleloggerparameters:NoSummary"
],
"problemMatcher": "$msCompile"
Expand Down Expand Up @@ -48,6 +49,12 @@
"default": "10.0.0",
"description": "The local version to build."
},
{
"id": "pluginNames",
"type": "promptString",
"default": "ReleaseExporter;RelocationPlus;ConfigurationHell",
"description": "The plugins to build."
},
// Open the command prompt (by pressing F1), remove the ">" and paste in "ext install augustocdias.tasks-shell-input" and hit enter.
{
"id": "commit",
Expand Down
5 changes: 1 addition & 4 deletions Shoko.CLI/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,7 @@ public static async Task Main()

try
{
var settingsProvider = new SettingsProvider(logFactory.CreateLogger<SettingsProvider>());
settingsProvider.LoadSettings();
Utils.SettingsProvider = settingsProvider;
var startup = new Startup(logFactory.CreateLogger<Startup>(), settingsProvider);
var startup = new Startup(logFactory);
startup.AboutToStart += (_, args) => AddEventHandlers(args.ServiceProvider);
await startup.Start();
await startup.WaitForShutdown();
Expand Down
4 changes: 2 additions & 2 deletions Shoko.Models/Client/CL_DuplicateFile.cs
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ public class CL_DuplicateFile

public int File1VideoLocalPlaceID { get; set; }
public int File2VideoLocalPlaceID { get; set; }
public ImportFolder ImportFolder1 { get; set; }
public ImportFolder ImportFolder2 { get; set; }
public CL_ImportFolder ImportFolder1 { get; set; }
public CL_ImportFolder ImportFolder2 { get; set; }
}
}
13 changes: 13 additions & 0 deletions Shoko.Models/Client/CL_ImportFolder.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@

namespace Shoko.Models.Client;

public class CL_ImportFolder
{
public int ImportFolderID { get; set; }
public int ImportFolderType { get; set; }
public string ImportFolderName { get; set; }
public string ImportFolderLocation { get; set; }
public int IsWatched { get; set; }
public int IsDropSource { get; set; }
public int IsDropDestination { get; set; }
}
18 changes: 15 additions & 3 deletions Shoko.Models/Client/CL_VideoLocal.cs
Original file line number Diff line number Diff line change
@@ -1,17 +1,29 @@
using System;
using System.Collections.Generic;
using Shoko.Models.PlexAndKodi;
using Shoko.Models.Server;

namespace Shoko.Models.Client
{
public class CL_VideoLocal : VideoLocal
public class CL_VideoLocal
{
public int VideoLocalID { get; set; }
public string Hash { get; set; }
public string CRC32 { get; set; }
public string MD5 { get; set; }
public string SHA1 { get; set; }
public int HashSource { get; set; }
public long FileSize { get; set; }
public int IsIgnored { get; set; }
public DateTime DateTimeUpdated { get; set; }
public DateTime DateTimeCreated { get; set; }
public DateTime? DateTimeImported { get; set; }
public int IsVariation { get; set; }
public string FileName { get; set; }
public int IsWatched { get; set; }
public DateTime? WatchedDate { get; set; }
public long ResumePosition { get; set; }
public List<CL_VideoLocal_Place> Places { get; set; }
public Media Media { get; set; }
public long Duration { get; set; }
}
}
}
9 changes: 7 additions & 2 deletions Shoko.Models/Client/CL_VideoLocal_Place.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,14 @@

namespace Shoko.Models.Client
{
public class CL_VideoLocal_Place : VideoLocal_Place
public class CL_VideoLocal_Place
{
public ImportFolder ImportFolder { get; set; }
public int VideoLocal_Place_ID { get; set; }
public int VideoLocalID { get; set; }
public string FilePath { get; set; }
public int ImportFolderID { get; set; }
public int ImportFolderType { get; set; }
public CL_ImportFolder ImportFolder { get; set; }

}
}
169 changes: 64 additions & 105 deletions Shoko.Models/Enums/AniDB.cs
Original file line number Diff line number Diff line change
@@ -1,113 +1,72 @@
using System;

namespace Shoko.Models.Enums;

namespace Shoko.Models.Enums
public enum AniDBFileDeleteType
{
public enum AiringState
{
All = 0,
StillAiring = 1,
FinishedAiring = 2,
}

public enum AniDBFileDeleteType
{
Delete = 0,
DeleteLocalOnly = 1,
MarkDeleted = 2,
MarkExternalStorage = 3,
MarkUnknown = 4,
MarkDisk = 5
}

public enum AniDBFile_State
{
Unknown,
HDD,
Disk,
Deleted,
Remote
}

[Flags]
public enum AniDBFileFlags
{
None = 0,
FILE_CRCOK = 1, //file matched official CRC (displayed with green background in AniDB)
FILE_CRCERR = 2, // file DID NOT match official CRC (displayed with red background in AniDB)
FILE_ISV2 = 4, // file is version 2
FILE_ISV3 = 8, // file is version 3
FILE_ISV4 = 16, // file is version 4
FILE_ISV5 = 32, // file is version 5
FILE_UNC = 64, // file is uncensored
FILE_CEN = 128, // file is censored
FILE_CHAPTERED = 4096 // file is chaptered, 0 means both not set and false
}

public enum AniDBUpdateType
{
None = 0,
UDPBan = 1,
HTTPBan = 2,
Invalid_Session = 3,
Overload_Backoff = 4,
WaitingOnResponse = 5,
}
Delete = 0,
DeleteLocalOnly = 1,
MarkDeleted = 2,
MarkExternalStorage = 3,
MarkUnknown = 4,
MarkDisk = 5
}

public enum AniDBRecommendationType
{
ForFans = 1,
Recommended = 2,
MustSee = 3,
}
public enum AniDBFile_State
{
Unknown,
HDD,
Disk,
Deleted,
Remote
}

public enum AniDBVoteType
{
Anime = 1,
AnimeTemp = 2,
Group = 3,
Episode = 4
}
public enum AniDBVoteType
{
Anime = 1,
AnimeTemp = 2,
Group = 3,
Episode = 4
}

public enum AniDB_ResourceLinkType
{
ANN = 1,
MAL = 2, // MAL ID, there may be more than one
AnimeNFO = 3, // Dead site.
Site_JP = 4, // Official Japanese Site
Site_EN = 5, // Official English Site
Wiki_EN = 6, // wikipedia.com
Wiki_JP = 7, // wikipedia.jp
Syoboi = 8, // Airing Schedule (Japanese site)
ALLCinema = 9,
Anison = 10,
DotLain = 11, // .lain (JP VA and anime site)
VNDB = 14, // The Visual Novel Database, for related VN game, if any.
Crunchyroll = 28, // Series page, not episodes
Amazon = 32, // amazon.com
Funimation = 34, // See Crunchyroll comment ☝
Bangumi = 38, // Japanese site
HiDive = 42, // Streaming service, series page.
}
public enum AniDB_ResourceLinkType
{
ANN = 1,
MAL = 2, // MAL ID, there may be more than one
AnimeNFO = 3, // Dead site.
Site_JP = 4, // Official Japanese Site
Site_EN = 5, // Official English Site
Wiki_EN = 6, // wikipedia.com
Wiki_JP = 7, // wikipedia.jp
Syoboi = 8, // Airing Schedule (Japanese site)
ALLCinema = 9,
Anison = 10,
DotLain = 11, // .lain (JP VA and anime site)
VNDB = 14, // The Visual Novel Database, for related VN game, if any.
Crunchyroll = 28, // Series page, not episodes
Amazon = 32, // amazon.com
Funimation = 34, // See Crunchyroll comment ☝
Bangumi = 38, // Japanese site
HiDive = 42, // Streaming service, series page.
}

public enum AnimeType
{
Unknown = -1, // Not on AniDB or not yet assigned a type.
Movie = 0,
OVA = 1,
TVSeries = 2,
TVSpecial = 3,
Web = 4,
Other = 5,
MusicVideo = 6,
}
public enum AnimeType
{
Unknown = -1, // Not on AniDB or not yet assigned a type.
Movie = 0,
OVA = 1,
TVSeries = 2,
TVSpecial = 3,
Web = 4,
Other = 5,
MusicVideo = 6,
}

public enum EpisodeType
{
Episode = 1,
Credits = 2,
Special = 3,
Trailer = 4,
Parody = 5,
Other = 6
}
public enum EpisodeType
{
Episode = 1,
Credits = 2,
Special = 3,
Trailer = 4,
Parody = 5,
Other = 6
}
10 changes: 0 additions & 10 deletions Shoko.Models/Enums/AzureLinkType.cs

This file was deleted.

14 changes: 0 additions & 14 deletions Shoko.Models/Enums/CrossRefType.cs

This file was deleted.

Loading