Skip to content

to prevent file cleaning, touch cache, its link, md5 and id, resolves #1282 #1367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
13 changes: 11 additions & 2 deletions Bourreau/lib/bourreau_system_checks.rb
Original file line number Diff line number Diff line change
Expand Up @@ -289,15 +289,24 @@ def self.a100_ensure_dp_cache_symlink_exists #:nodoc:
myself = RemoteResource.current_resource
gridshare_dir = myself.cms_shared_dir
cache_dir = myself.dp_cache_dir
sym_path = "#{gridshare_dir}/#{DataProvider::DP_CACHE_SYML}"

return unless Dir.exists?(gridshare_dir) && Dir.exists?(cache_dir)

#----------------------------------------------------------------------------
puts "C> Making sure the grid share directory has a symlink to the data provider cache..."
#----------------------------------------------------------------------------

sym_path = "#{gridshare_dir}/#{DataProvider::DP_CACHE_SYML}"
return if File.symlink?(sym_path) && File.realpath(sym_path) == File.realpath(cache_dir)
# update the work directory timestamp to counter cluster deletion policies
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this so complicated?!??

The original code's logic was:

return if symlink_is_ok
create symlink

The new code should simply be instead:

if symlink_is_ok
  touch symlink
  return
end
create symlink

But you did a complex set of conditions including checking the return status of system():

touch_but_not_create symlink
ret=system(touch symlink)
return if ret is 0 and symlink_is_ok
create_symlink

Why is make it so complicated?

Copy link
Contributor Author

@MontrealSergiy MontrealSergiy Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make code work even if touch fails. But ok, can roll back that check.

FileUtils.touch(gridshare_dir, verbose: true, nocreate: true)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you leave the verbose and nocreate option there? Did you check the boot logs? I now see this crud:

C> Trying to see if there are any spurious task work directories...
C>      - No spurious task work directories detected.
C> Making sure the grid share directory has a symlink to the data provider cache...
touch -c /Users/prioux/LargeData/BourreauGridshare
C> Ensuring git commits for tasks classes are pre-cached...
C> Making sure the portal is forwarding a SSH agent to us...
C>      - Found a forwarded agent on SOCK=/tmp/ssh-8bZbwKlpmA/agent.13854


if File.symlink?(sym_path) && File.realpath(sym_path) == File.realpath(cache_dir)
# touch -h updates the timestamp of the symlink itself not the file it point to
# it works on major modern Linux and MacOS, but might have problems with older OS,
# such as Big Sur
system("touch", "-h", sym_path)
return
end

File.unlink(sym_path) if File.exists?(sym_path)
File.symlink(cache_dir, sym_path)
Expand Down
13 changes: 13 additions & 0 deletions BrainPortal/app/models/data_provider.rb
Original file line number Diff line number Diff line change
Expand Up @@ -1206,6 +1206,19 @@ def self.create_cache_md5
end
end

# This method updates timestamps of the +DataProvider+ cache dir +DP_Cache+, its md5 token +DP_Cache_Key.md5+ and
# revision info +DP_Cache_Rev.id+.
# Some systems are configured with disk allocations where files older than N days are erased automatically.
# So CBRAIN reset timestamps of few administrative files or dir using the touch command.
def self.touch_cache_md5
myself = RemoteResource.current_resource
cache_dir = myself.dp_cache_dir
dp_cache_id = File.join cache_dir, DataProvider::DP_CACHE_ID_FILE
dp_cache_md5 = File.join cache_dir, DataProvider::DP_CACHE_MD5_FILE

FileUtils.touch([cache_dir, dp_cache_id, dp_cache_md5], verbose: true, nocreate: true)
end

# This method returns the revision DateTime of the last time
# the caching system was initialized. If the revision
# number is unknown, then a string value of "Unknown" is returned
Expand Down
18 changes: 18 additions & 0 deletions BrainPortal/lib/cbrain_system_checks.rb
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,24 @@ def self.a050_check_data_provider_cache_wipe #:nodoc:
end
end

# prevents archiving or deleting of administrative files and top directories related to
# cache ( gridshare touching is performed by a Bourreau system check method )
def self.a060_ensure_top_cache_dir_will_not_be_deleted #:nodoc:

#-----------------------------------------------------------------------------
puts "C> Updating DataProvider cache administrative files and directories"
#-----------------------------------------------------------------------------

cache_root = DataProvider.cache_rootdir rescue nil
# Need to perform a `to_s` due to a strange behaviour of `blank?`
# on `Pathname` (if a content of a `Pathname` is empty it will return true)
if cache_root.to_s.blank?
puts "C> \t- SKIPPING! No cache root directory yet configured!"
return
end

DataProvider.touch_cache_md5
end


def self.a080_ensure_set_starttime_revision #:nodoc:
Expand Down