Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for CEF 120 and newer #3149

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from

Conversation

vldevel
Copy link
Contributor

@vldevel vldevel commented Nov 25, 2024

LL's viewer is currently using CEF 118, which is riddled with security holes and shall be updated for the current CEF version. Problem: with CEF 120, a new caching scheme was introduced which forbids sharing a same cache between several CEF instances, like what we did so far, and which allowed to keep Cookies in sync between instances.

For my viewer and under Linux, I came up with an easy "solution" (more like a hack), which involved pre-populating each CEF instance cache with a symbolic link to a central Cookies store, and it does work just like it used to do with older CEF versions (i.e. with a shared Cookies file between instances).
While this solution/hack would also likely work with macOS (after we will have a Dullahan/CEF 131 available for it), it however won't work under Windows (Callum Linden and I tried, but Windows is not a POSIX system...).

During our email-exchanges/brainstorming with Callum, I came up with the idea of a Cookies store "merger". The principle is that each new CEF session sees its cache primed by the CEF plugin with a copy of a central Cookies store before launch, and that on exit of that CEF session, the plugin syncs the central Cookies store file with the session updated Cookies file (adding new cookies found in the latter and updating cookies already in the central database but that got changed during the session).

This commit implements this mechanism. It has been in use in the Cool VL Viewer releases for over a couple of weeks already, without any bug report or user complaint about it.

Implementation notes:

  1. The "merger" code requires sqlite3, and not knowing how to create a 3p-libsqlite3 repository, this commit links (via autobuild.xml) to personal builds of libsqlite3. You will want to replace those links with LL-built packages. For your 3p-libsqlite3, you can reuse the build scripts I wrote, available here:
    http://sldev.free.fr/libraries/sources/cvlv-libsqlite3-20241112.tar.bz2

  2. The cache structure changed (see my comment in media_plugin_cef.cpp for "set_user_data_path", around line 1008), but this should not be an issue.

  3. The session caches are now temporary and get erased at each session exit (after the added/changed Cookies have been saved into the central Cookies store); sadly, under Windows (this is not happening under Linux), CEF keeps writing a few files back into the destroyed temporary cache after the plugin destructor returns (see my comment at the end of ~MediaPluginCEF() around line 263), meaning the temp cache directory is recreated and will contain a few remnants.
    This is harmless but in my viewer I added code to clear everything but the "Cookies*" files into cef_cache/ (i.e. clearing all temp caches remnants in there) on viewer launch (for the "anonymous" cache used at login screen) and after login (for the per-account cache); since there is no equivalent in LL's code to my method in my improved/expanded/ rewritten LLDiriterator for doing this, I did not "bother" adding such code to this commit. Let me know if you are interested in my LLDiriterator implementation. Alternatively, an ad hoc cleaning up code could be implemented.

  4. So far, only Windows is concerned by this commit since I only have the link for a Dullahan/CEF131 pre-built library Callum Linden kindly passed to me. macOS and Linux will keep using CEF 118 for now (the plugin will automatically compile for it) and this commit should not break anything for them.
    But when LL will come up with a Linux pre-built library, the code is ready for it (just change the link to dullahan in autobuild.xml and the CEF plugin will automatically compile for CEF 131); the same might be true for macOS, however a few of things will need to be checked:
    a. Where the "Cookies" file lives in the macOS CEF cache tree.
    b. Whether there is a crypto key stored in "Local State" (like for
    Windows) or not (like for Linux) that prompts saving that file as
    well or not.
    c. What marker file to watch for at CEF exit, before attempting to
    merge the Cookies
    I do not have a MAC, so I cannot check for the above. However, all it will take will be to adjust the corresponding #defines in the media_plugin_cef.cpp file for LL_DARWIN case; see my comments in that line 84 and line 101. To be safe, I added a #warning for macOS and CEF 120 or newer.

LL's viewer is currently using CEF 118, which is riddled with security
holes and shall be updated for the current CEF version.
Problem: with CEF 120, a new caching scheme was introduced which forbids
sharing a same cache between several CEF instances, like what we did so
far, and which allowed to keep Cookies in sync between instances.

For my viewer and under Linux, I came up with an easy "solution" (more
like a hack), which involved pre-populating each CEF instance cache with
a symbolic link to a central Cookies store, and it does work just like
it used to do with older CEF versions (i.e. with a shared Cookies file
between instances).
While this solution/hack would also likely work with macOS (after we
will have a Dullahan/CEF 131 available for it), it however won't work
under Windows (Callum Linden and I tried, but Windows is not a POSIX
system...).

During our email-exchanges/brainstorming with Callum, I came up with the
idea of a Cookies store "merger". The principle is that each new CEF
session sees its cache primed by the CEF plugin with a copy of a central
Cookies store before launch, and that on exit of that CEF session, the
plugin syncs the central Cookies store file with the session updated
Cookies file (adding new cookies found in the latter and updating
cookies already in the central database but that got changed during the
session).

This commit implements this mechanism. It has been in use in the
Cool VL Viewer releases for over a couple of weeks already, without
any bug report or user complaint about it.

Implementation notes:
====================
1. The "merger" code requires sqlite3, and not knowing how to create a
3p-libsqlite3 repository, this commit links (via autobuild.xml) to
personal builds of libsqlite3. You will want to replace those links
with LL-built packages. For your 3p-libsqlite3, you can reuse the
build scripts I wrote, available here:
http://sldev.free.fr/libraries/sources/cvlv-libsqlite3-20241112.tar.bz2

2. The cache structure changed (see my comment in media_plugin_cef.cpp
for "set_user_data_path", around line 1008), but this should not be an
issue.

3. The session caches are now temporary and get erased at each session
exit (after the added/changed Cookies have been saved into the central
Cookies store); sadly, under Windows (this is not happening under
Linux), CEF keeps writing a few files back into the destroyed temporary
cache after the plugin destructor returns (see my comment at the end of
~MediaPluginCEF() around line 263), meaning the temp cache directory is
recreated and will contain a few remnants.
This is harmless but in my viewer I added code to clear everything _but_
the "Cookies*" files into cef_cache/ (i.e. clearing all temp caches
remnants in there) on viewer launch (for the "anonymous" cache used at
login screen) and after login (for the per-account cache); since there
is no equivalent in LL's code to my method in my improved/expanded/
rewritten LLDiriterator for doing this, I did not "bother" adding such
code to this commit. Let me know if you are interested in my
LLDiriterator implementation. Alternatively, an ad hoc cleaning up code
could be implemented.

4. So far, only Windows is concerned by this commit since I only have
the link for a Dullahan/CEF131 pre-built library Callum Linden kindly
passed to me. macOS and Linux will keep using CEF 118 for now (the
plugin will automatically compile for it) and this commit should not
break anything for them.
But when LL will come up with a Linux pre-built library, the code is
ready for it (just change the link to dullahan in autobuild.xml and the
CEF plugin will automatically compile for CEF 131); the same *might* be
true for macOS, however a few of things will need to be checked:
 a. Where the "Cookies" file lives in the macOS CEF cache tree.
 b. Whether there is a crypto key stored in "Local State" (like for
    Windows) or not (like for Linux) that prompts saving that file as
    well or not.
 c. What marker file to watch for at CEF exit, before attempting to
    merge the Cookies
I do not have a MAC, so I cannot check for the above. However, all it
will take will be to adjust the corresponding #defines in the
media_plugin_cef.cpp file for LL_DARWIN case; see my comments in that
line 84 and line 101. To be safe, I added a #warning for macOS and CEF
120 or newer.
@marchcat
Copy link
Contributor

Thank you for the contribution!

@marchcat
Copy link
Contributor

Something is wrong with the sqlite package:

-- Installing libsqlite3...
ERROR: nonexistent license_file for libsqlite3: include/sqlite3.h
For more information: try re-running your command with --verbose or --debug
CMake Error at cmake/Prebuilt.cmake:57 (message):
  Failed to download or unpack prebuilt 'libsqlite3'.  Process returned 1.
Call Stack (most recent call first):
  cmake/SQLite3.cmake:9 (use_prebuilt_binary)
  media_plugins/cef/CMakeLists.txt:14 (include)

also, there are tab indents:

indra/newview/llviewermedia.cpp: 2 lines starting with tabs found

@vldevel
Copy link
Contributor Author

vldevel commented Nov 25, 2024

Something is wrong with the sqlite package:

-- Installing libsqlite3...
ERROR: nonexistent license_file for libsqlite3: include/sqlite3.h
For more information: try re-running your command with --verbose or --debug
CMake Error at cmake/Prebuilt.cmake:57 (message):
  Failed to download or unpack prebuilt 'libsqlite3'.  Process returned 1.
Call Stack (most recent call first):
  cmake/SQLite3.cmake:9 (use_prebuilt_binary)
  media_plugins/cef/CMakeLists.txt:14 (include)

I had to disable the licenses file check in LL's autobuild's python script to get it working...

Like I explained in the implementation notes, I do not know how to build your 3p-stuff, and sqlite3 being public domain, it does not have a license, so I gave the include/sqlite3.h header file as the license file (where it is written there is no license). It's rather "strange" autobuild cannot "find" the packaged include file or does not accept it as a license file !!!

Not sure what to do about this... I lost enough time (and my nerves) on this stupid 'autobuild' already, sorry. :-/

also, there are tab indents:
indra/newview/llviewermedia.cpp: 2 lines starting with tabs found

Will fix that.

I really, thoroughly hate your autobuild system, LL, you know... :-(

I lost another full hour to find out it did not like tar files created
from "." (current dirctory). You must instead use the "*" wild card to
include everything inside the curret directory, else autobuild cannot
find its "meta data" and license files !
@vldevel
Copy link
Contributor Author

vldevel commented Nov 25, 2024

Finally found out what autobuild did not like in my pre-built packages...
Fix committed.

@vldevel
Copy link
Contributor Author

vldevel commented Dec 12, 2024

Callum told me in an email he "was moved to a different (non Viewer) team".

So, the reviewer for this PR should be changed for someone else...

@marchcat marchcat self-requested a review January 3, 2025 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants