Renewal of payload proxies

[arrabito]

Dear all,

Could you please tell me which mechanism is responsible for the renewal of payload proxies, is it the ProxyRenewalAgent?

At one site payload proxies get expired after 12 hours :

PEM file has expired /tmp/tmpnmrlzdpo is not valid after 2025-10-29 21:43:00

while at another site they don't.

So I guess that there is some different configuration at the 2 sites.

Looking at the pilot logs I found that at one site (where proxies get expired) there is the following variable set:

MYPROXY_SERVER

while at the other is not. Could it be the cause of the renewal problem?

As an additional information, but I don't know if it's relevant, we have the following configuration for the ProxyRenewalAgent:

ProxyRenewalAgent
        {
          PollingTime = 900
          #Email to use as a sender for the expiration reminder
          MailFrom = [[email protected]](mailto:[email protected])
          #Renew, or not, the proxies stored on the MyProxy service
          UseMyProxy = False
          MinimumLifeTime = 3600
          RenewedLifeTime = 54000
        }

Thank you for your help,

Luisa

[fstagni]

Hi,
the ProxyRenewalAgent, despite its name, does not renew proxies (name is something historical), just purge old ones.
MYPROXY_SERVER is another historical thing that is completely unsupported by now, so do not look at it.

The ComputingElement asks for proxy renewal, with these settings:

        self.minProxyTime = gConfig.getValue("/Registry/MinProxyLifeTime", 10800)  # secs
        self.defaultProxyTime = gConfig.getValue("/Registry/DefaultProxyLifeTime", 43200)  # secs
        self.proxyCheckPeriod = gConfig.getValue("/Registry/ProxyCheckingPeriod", 3600)  # secs

Modify at will.

[arrabito]

Hi @fstagni

Thank you for the explanations. I will try to change this value /Registry/DefaultProxyLifeTime.

But I don't really understand how at one site there are successfull jobs lasting more than 12 hours while at another I see the PEM expired message after 12 hours....

Any idea?

[fstagni]

Are these HtCondorCE or ARC CE sites? Or SSH?

[arrabito]

They are both HtCondorCE.

Then we also have ARC CE and SSH sites. Is there anything to know about the last 2?

[fstagni]

Actually, IIUC the MYPROXY_SERVER identifies the myproxy server used for proxy renewal in the HTCondor world. That might have settings such that expires the proxy after 12 hours.
ARC and HTCondor will try to manage your proxies, for SSH instead it's all internal to DIRAC.

[arrabito]

I see. So let's see if my change in /Registry/DefaultProxyLifeTime is enough. I will let you know.

[arrabito]

Hi @fstagni I've set:

/Registry/DefaultProxyLifeTime = 86400

but I still see proxy generated with 12 hours of lifetime, see below:

2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Requesting proxy', 'for /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa [email protected]@ctao_prod
2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Found Job LogLevel JDL parameter with value: INFO
2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Submitting JobWrapper Job21375244 to PoolCE
2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Job being submitted (DIRAC JobID: 21375244; Task ID: 0)
2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/Task-0 INFO: Payload proxy information:
subject      : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa [email protected]/CN=8401460194/CN=6460439606/CN=3911717654/CN=8853180364
issuer       : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa [email protected]/CN=8401460194/CN=6460439606/CN=3911717654
identity     : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa [email protected]
timeleft     : 11:59:58
DIRAC group  : ctao_prod
path         : /tmp/tmpl18bwej3
username     : arrabito
properties   : NormalUser, PrivateLimitedDelegation, ProductionManagement, JobAdministrator
VOMS         : True
VOMS fqan    : ['/ctao/Role=production']

Making again jobs fail.

Any idea?

Thank you.

[fstagni]

Hi, few questions:

• This is DIRAC v8 or v9? There have been changes between the 2 versions in this respect.
• what do you see in the log of the pilots going to the other site (where it does not fail?)
• in the jobAgent I see self.defaultProxyLength = 86400 * 5 which is 5 days, so there's something wrong.

[arrabito]

Hi @fstagni

* This is DIRAC v8 or v9? There have been changes between the 2 versions in this respect.

Sorry, I've forgot to specify that it's v8.0.71

* what do you see in the log of the pilots going to the other site (where it does not fail?)

I see the same duration of 12 hours proxy:

2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Requesting proxy', 'for /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa [email protected]@ctao_prod
....
timeleft     : 11:59:58

but when jobs have a duration of more than 12 hours, I don't see the error : "Proxy has expired" as at PIC, and they finish well.

* in the jobAgent I see `self.defaultProxyLength = 86400 * 5` which is 5 days, so there's something wrong.

that's strange indeed

[fstagni]

Would you by chance be able to try with v9?

[aldbr]

Hi @arrabito, here we are talking about the user proxy downloaded by the JobAgent once it fetches a job, aren't we?

I see that the user proxy could be renewed in ComputingElement._monitorProxy() IIUC:

DIRAC/src/DIRAC/Resources/Computing/ComputingElement.py

Lines 359 to 461 in 106cca5

	def _monitorProxy(self, payloadProxy=None):
	"""Base class for the monitor and update of the payload proxy, to be used in
	derived classes for the basic renewal of the proxy, if further actions are
	necessary they should be implemented there
	:param str payloadProxy: location of the payload proxy file
	:returns: S_OK(filename)/S_ERROR
	"""
	if not payloadProxy:
	return S_ERROR("No payload proxy")
	# This will get the pilot proxy
	ret = getProxyInfo()
	if not ret["OK"]:
	pilotProxy = None
	else:
	pilotProxy = ret["Value"]["path"]
	self.log.notice("Pilot Proxy:", pilotProxy)
	retVal = getProxyInfo(payloadProxy)
	if not retVal["OK"]:
	self.log.error("Could not get payload proxy info", retVal)
	return retVal
	self.log.verbose(f"Payload Proxy information:\n{formatProxyInfoAsString(retVal['Value'])}")
	payloadProxyDict = retVal["Value"]
	payloadSecs = payloadProxyDict["chain"].getRemainingSecs()["Value"]
	if payloadSecs > self.minProxyTime:
	self.log.verbose("No need to renew payload Proxy")
	return S_OK()
	# if there is no pilot proxy, assume there is a certificate and try a renewal
	if not pilotProxy:
	self.log.info("Using default credentials to get a new payload Proxy")
	return gProxyManager.renewProxy(
	proxyToBeRenewed=payloadProxy,
	minLifeTime=self.minProxyTime,
	newProxyLifeTime=self.defaultProxyTime,
	proxyToConnect=pilotProxy,
	)
	# if there is pilot proxy
	retVal = getProxyInfo(pilotProxy)
	if not retVal["OK"]:
	return retVal
	pilotProxyDict = retVal["Value"]
	if "groupProperties" not in pilotProxyDict:
	self.log.error("Invalid Pilot Proxy", "Group has no properties defined")
	return S_ERROR("Proxy has no group properties defined")
	pilotProps = pilotProxyDict["groupProperties"]
	# if running with a pilot proxy, use it to renew the proxy of the payload
	if Properties.PILOT in pilotProps or Properties.GENERIC_PILOT in pilotProps:
	self.log.info("Using Pilot credentials to get a new payload Proxy")
	return gProxyManager.renewProxy(
	proxyToBeRenewed=payloadProxy,
	minLifeTime=self.minProxyTime,
	newProxyLifeTime=self.defaultProxyTime,
	proxyToConnect=pilotProxy,
	)
	# if we are running with other type of proxy check if they are for the same user and group
	# and copy the pilot proxy if necessary
	self.log.info("Trying to copy pilot Proxy to get a new payload Proxy")
	pilotProxySecs = pilotProxyDict["chain"].getRemainingSecs()["Value"]
	if pilotProxySecs <= payloadSecs:
	errorStr = "Pilot Proxy is not longer than payload Proxy"
	self.log.error(errorStr)
	return S_ERROR(f"Can not renew by copy: {errorStr}")
	# check if both proxies belong to the same user and group
	pilotDN = pilotProxyDict["chain"].getIssuerCert()["Value"].getSubjectDN()["Value"]
	retVal = pilotProxyDict["chain"].getDIRACGroup()
	if not retVal["OK"]:
	return retVal
	pilotGroup = retVal["Value"]
	payloadDN = payloadProxyDict["chain"].getIssuerCert()["Value"].getSubjectDN()["Value"]
	retVal = payloadProxyDict["chain"].getDIRACGroup()
	if not retVal["OK"]:
	return retVal
	payloadGroup = retVal["Value"]
	if pilotDN != payloadDN or pilotGroup != payloadGroup:
	errorStr = "Pilot Proxy and payload Proxy do not have same DN and Group"
	self.log.error(errorStr)
	return S_ERROR(f"Can not renew by copy: {errorStr}")
	if pilotProxyDict.get("hasVOMS", False):
	return pilotProxyDict["chain"].dumpAllToFile(payloadProxy)
	attribute = Registry.getVOMSAttributeForGroup(payloadGroup)
	vo = Registry.getVOMSVOForGroup(payloadGroup)
	retVal = VOMS().setVOMSAttributes(pilotProxyDict["chain"], attribute=attribute, vo=vo)
	if not retVal["OK"]:
	return retVal
	chain = retVal["Value"]
	return chain.dumpAllToFile(payloadProxy)

Can you spot the logs in your pilot outputs to compare them? Such as:

- Pilot Proxy:
- No need to renew payload Proxy
- Using default credentials to get a new payload Proxy