⚠ CMS SST — Daily Site Report

Hello,

Now that the 2026 data taking has started, we are reviewing the state of the storage
pledges on the LHCb T1 and T2 sites.

Earlier this year, we were informed that (with a few exceptions) the sites would have
the pledged disk and tape resources available to the experiment, and we hope you
can confirm that.

1 - Status of disk pledges

PIC pledges 5855 TB of disk space for 2026, which includes a recent addition of 500 TB.

However, the Storage Resource Record (SRR) advertises 5393.6 TB [1].

Do you know what explains that discrepancy?

2 - Status of tape pledges

The site pledges 11685 TB of tape space for 2026. Are these resources available
to LHCb already?

Thanks in advance for your efforts to make these crucial resources available
to us!

best regards, Jan van Eldik / LHCb Compute project lead
[1]
+-------+--------------+-----------+-----------+----------+
| Site | Share | Size | Used | Fraction |
+-------+--------------+-----------+-----------+----------+
| PIC | LHCb_BUFFER | 113.6 | 113.6 | 100.0% |
| PIC | LHCb-Disk | 4900.0 | 4274.1 | 87.2% |
| PIC | LHCb-Tape | 300.0 | 130.7 | 43.6% |
| PIC | LHCb_USER | 80.0 | 62.1 | 77.7% |
+-------+--------------+-----------+-----------+----------+

Hi Jan,

The discrepancy in the disk numbers is due to the fact that the last increment has not yet been assigned.
We'll do it as soon as possible.
Regarding the tape assignment, we are running a bit behind schedule because the installation of the new library was only recently completed, and we are currently conducting validation tests.
Once these tests are finished, we plan to migrate to CTA first (from Enstore), and then we will proceed with the assignments in the new robot.

Don't hesitate to contact us for any question.

Regards,
Elena

Thanks for the quick reply Elena, a few quick questions:
> The discrepancy in the disk numbers is due to the fact that the last increment has not yet been assigned.> We'll do it as soon as possible.
Thanks. Would that be a matter of days? Or of weeks?
Similar questions for tape:
> we plan to migrate to CTA first (from Enstore), and then we will proceed with the assignments in the new robot.
Can you give us an idea of the timescale? Also, will we be able to access the existing tape resources (ie. the 2025 pledge of 8766 TB, or maybe a bit more?) until the migration happens?

Thanks for the information, this will help us to plan the data distribution of the fresh 2026 data.

best, Jan

Dear site admins,

The HTCondorCE v23 series (and older) became unsopported and the endpoints running it should be either decommissioned or upgraded to 24.0.x series.

You received this ticket either because you provide at least one HTCondorCE endpoint out of support or because you provide HTCondorCE endpoint(s) but we couldn't determine the version by looking into the BDII.

If you are running a supported version of HTCondor, please let us know which one is, make sure that the endpoints are properly published into the BDII (which will make it easier to carry on activities like this one), and then close the ticket.

Instead, if you are running an unsupported version, we ask you to upgrade it as soon as possible.
In the UMD repository you can find HTCondor-CE 24.0.2 and HTCondor 24.0.14, which is the minimum version that we recommend.
Please check the full release notes of the 24.0.x series (https://htcondor.readthedocs.io/en/latest/version-history/lts-versions-24-0.html) and pay attention to the differences between v23.0.x and v24.0.x in terms of settings and features (for example the different syntax used for the SSL mapping).
Please read carefully the documentation before the upgrade: all the changes with the upgrade must be applied manually, in particular the changes to the new syntax for the SSL mapping.

The quick configuration guide for HTCondor24 created by WLCG can be useful for the upgrade process: https://twiki.cern.ch/twiki/bin/view/LCG/MiniHTCv24EL9

Thanks for your collaboration,
EGI Operations

Hi Alessandro,
We already have the version 24:
htcondor-ce-24.0.2-1.el9.noarch
condor-24.0.11-1.el9.x86_64
Where did you find we have version 23?
Thank you,
Vanessa

Hi Vanessa,
thanks for the reply. When I looked for the version you run, I wasn't able to find it because the endpoint is not published in the BDII, so I had to open the ticket anyway. If you manage to configure the infoprovider and publish the CE in the bdii, it would be helpful.

Cheers,
Alessandro

Hi Vanessa,
actually I found the following CE with an unsupported version:

- cccondorcm01.in2p3.fr: 9.0.20
- cccondorcm02.in2p3.fr: 23.10.1

They are registered in GOCDB and the version information is published in the BDII.

Cheers,
Alessandro

Hi Alessandro,

Those are Central Managers (cm), and we used to publish long time ago in the CM (only in one point) looks like our BDIIs are not in good shape. I will check.

Thank you,
Vanessa

For the last 2 weeks we have seen xrootd token access fail on xrootd-archive.cr.cnaf.infn.it for DUNE.the root port advertises the ZTN protocol as it should, but we get error:
xrdcp root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt ./infnroot2.txt

sec_PM: Loaded ztn protocol object from libXrdSecztn.so

sec_PM: Using ztn protocol, args='0:4096:'
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3010] Unable to open /dune/testpro/bb/7f/awt-download-2023-03-07-01.txt; permission denied (source)

It is the same error with root:// and roots:// protocol. davs:// protocol is not affected.

Please investigate. This used to work up until a couple weeks ago. Above test was made Thursday. 26 Mar at 11:01 US/Central time = 17:01 Central Europe time

{
"sub": "dunepro@fnal.gov",
"iss": "https://cilogon.org/dune",
"wlcg.ver": "1.0",
"aud": "https://wlcg.cern.ch/jwt/v1/any",
"acr": "https://refeds.org/profile/sfa",
"nbf": 1774531982,
"auth_time": 1774531981,
"scope": "storage.read:/resilient/jobsub_stage storage.create:/ compute.cancel compute.create compute.read storage.read:/ storage.create:/resilient/jobsub_stage storage.create:/persistent/jobsub/jobs storage.stage:/ compute.modify",
"exp": 1774542787,
"iat": 1774531987,
"wlcg.groups": [
"/dune/production",
"/dune"
],
"jti": "<redacted>"
}

Actually my earlier statement was not correct.. the davs:// protocol is not working either

davs://xfer-archive.cr.cnaf.infn.it:8443/dune-grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt

gfal-copy error: 13 (Permission denied) - Could not stat the source: Result (Neon): SSL handshake failed: Connection timed out during SSL handshake after 1 attempts

Hi,
is it possibile that the DN of the FNAL voms servers changed without us being notified about it?
We currently have the following LSC files for DUNE vo:
# cat voms1.fnal.gov.lsc
/DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Forward Discovery Group, LLC/CN=voms1.fnal.gov
/C=US/O=Internet2/CN=InCommon RSA IGTF Server CA 3

# cat voms2.fnal.gov.lsc
/DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Forward Discovery Group, LLC/CN=voms2.fnal.gov
/C=US/O=Internet2/CN=InCommon RSA IGTF Server CA 3

Could you please check?
Thanks,
lucia

That is the right DN for the VOMS servers.
X.509 proxy authentication is fine.
It is only token authentication that changed.
There was an update of the scitokens lib that went out in EPEL the week before last, maybe that changed something? (upgrading to 1.4).
Steve Timm

From: helpdesk@ggus.org <helpdesk@ggus.org>
Sent: Friday, March 27, 2026 3:57 AM
To: Steven Timm <timm@fnal.gov>
Subject: [GGUS-Ticket-ID: #1002202] "ASSIGNED" "NGI_IT" "Token authentication fails with xrootd on xrootd-archive.cr.cnaf.infn.it" (Updated)

[EXTERNAL] – This message is from an external sender

GGUS Helpdesk Notification
Ticket #1002202 "Token authentication fails with xrootd on xrootd-archive.cr.cnaf.infn.it"
was updated by Lucia Morganti on 2026-03-27 08:57 (UTC).

Hi,
is it possibile that the DN of the FNAL voms servers changed without us being notified about it?
We currently have the following LSC files for DUNE vo:

# cat voms1.fnal.gov.lsc
/DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Forward Discovery Group, LLC/CN=voms1.fnal.gov
/C=US/O=Internet2/CN=InCommon RSA IGTF Server CA 3

# cat voms2.fnal.gov.lsc
/DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Forward Discovery Group, LLC/CN=voms2.fnal.gov
/C=US/O=Internet2/CN=InCommon RSA IGTF Server CA 3

Could you please check?
Thanks,
lucia

Ticket is assigned to NGIs › NGI_IT.

Affected VO is dune.

https://helpdesk.ggus.eu/#ticket/zoom/1002202

You are receiving this because you were subscribed via Mailing List (Notified mode on every change) in this ticket. | Manage your notification settings | EGI/WLCG

Ok, thanks.
If this is the right DN for the VOMS servers, then please the VO card should be updated accordingly:
https://operations-portal.egi.eu/vo/view/voname/dune#VOV_section
The fact that you find problems with both protocols would suggest the tokens have something wrong.
We did not change anything in our servers lately, both xfer-archive.cr.cnaf.infn.it and xrootd-archive.cr.cnaf.infn.it.
In xrootd-archive.cr.cnaf.infn.it we have the following scitoken rpms:
xrootd-scitokens-5.7.3-1.el9.x86_64
scitokens-cpp-1.3.0-1.el9.x86_64
In
xfer-archive.cr.cnaf.infn.it the following validator is used by StoRM
WebDAV:
https://github.com/italiangrid/storm-webdav/blob/develop/src/main/java/org/italiangrid/storm/webdav/oauth/validator/WlcgProfileValidator.java
Hope this helps,
lucia

IT is not straightforward for me to run the above code but I looked at it by examination and my tokens have all the fields that are being requested and they are of legal ranges.I'll add some debugging to the client and see if gives a clue.

Hi Steven,
sorry for the late reply.

I would like to start with the WebDAV endpoint.
Could you please try to run the following commands and share the output with us?

If you use oidc-agent to store the token:
$ export BEARER_TOKEN=$(oidc-token <dune>)
$ voms-proxy-destroy
$ unset X509_USER_PROXY
$ gfal-ls -vv davs://xfer-archive.cr.cnaf.infn.it:8443/dune-grid
$ rpm -qa | grep ca-policy
$ rpm -qa | grep gfal

Thanks,
Andrea

OK here you go
As you can see from the output, this test was actually successful.

------------------------------------------------------------------------------
<> htgettoken -i dune -r production -a htvaultprod.fnal.gov
Attempting to get token from https://htvaultprod.fnal.gov:8200 ... succeeded
Storing bearer token in /run/user/2904/bt_u2904
<> export BEARER_TOKEN=`cat /run/user/2904/bt_u2904`
<> voms-proxy-destroy

Proxy file doesn't exist or has bad permissions

<> id
uid=2904(timm) gid=9010(dune) groups=9010(dune),2038(duneatmexot),2042(duneadmintest),8080(dunecal),8085(dunehe),8086(dunele),8087(dunendsim),8089(dunefdsim),8090(dunenuint),8092(dunelbl),8094(dunebsm),8095(dunepd),8271(duneadmin),8660(ifadmin-mgr),9960(lbnf)
<> ls -l /tmp/x509up_u2904
ls: cannot access '/tmp/x509up_u2904': No such file or directory
<> unset X509_USER_PROXY
<> gfal-ls -vv davs://xfer-archive.cr.cnaf.infn.it:8443/dune-grid
INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success
INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success
INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success
INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success
INFO Davix: Unable to find a proxy or cert/key pair using either X509_USER_* variables or /tmp/x509up_u2904
INFO Davix: Unable to find a proxy or cert/key pair using either X509_USER_* variables or /tmp/x509up_u2904
INFO [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success
INFO Using bearer token for HTTPS request authorization
INFO Davix: > PROPFIND /dune-grid HTTP/1.1
> User-Agent: gfal2-util/1.9.1 gfal2/2.23.5 neon/0.0.29
> Keep-Alive:
> Connection: Keep-Alive
> TE: trailers
> Host: xfer-archive.cr.cnaf.infn.it:8443
> Depth: 0
> Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>

INFO Davix: < HTTP/1.1 207 Multi-Status
INFO Davix: < Date: Fri, 03 Apr 2026 14:05:24 GMT
INFO Davix: < Content-Type: application/xml; charset=utf-8
INFO Davix: < Content-Length: 722
INFO Davix: < Connection: keep-alive
INFO Davix: < Server: StoRM-WebDAV/1.12.0 (instance=tTFk)
INFO Davix: < DAV: 1
INFO Davix: < X-Content-Type-Options: nosniff
INFO Davix: < X-XSS-Protection: 0
INFO Davix: < Cache-Control: no-cache, no-store, max-age=0, must-revalidate
INFO Davix: < Pragma: no-cache
INFO Davix: < Expires: 0
INFO Davix: < X-Frame-Options: DENY
INFO Davix: <
INFO Using bearer token for HTTPS request authorization
INFO Davix: > PROPFIND /dune-grid HTTP/1.1
> User-Agent: gfal2-util/1.9.1 gfal2/2.23.5 neon/0.0.29
> TE: trailers
> Host: xfer-archive.cr.cnaf.infn.it:8443
> Depth: 1
> Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

I double-checked and the normal test I have been running myself with the davs URI works too,which is a gfal-copy.
davs://xfer-archive.cr.cnaf.infn.it:8443/dune-grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt

Copying davs://xfer-archive.cr.cnaf.infn.it:8443/dune-grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt [DONE] after 0s

<> httokendecode
{
"sub": "dunepro@fnal.gov",
"iss": "https://cilogon.org/dune",
"wlcg.ver": "1.0",
"aud": "https://wlcg.cern.ch/jwt/v1/any",
"acr": "https://refeds.org/profile/sfa",
"nbf": 1775225020,
"auth_time": 1775057711,
"scope": "storage.read:/resilient/jobsub_stage storage.create:/ compute.cancel compute.create compute.read storage.read:/ storage.create:/resilient/jobsub_stage storage.create:/persistent/jobsub/jobs storage.stage:/ compute.modify",
"exp": 1775235825,
"iat": 1775225025,
"wlcg.groups": [
"/dune/production",
"/dune"
],
"jti": "<redacted>"
}

Great,

gfal always tries to use the VOMS proxy before the token.
This was probably the cause of the problem.

So I think we can now move on to XRootD.

Could you also run:

xrdfs root://xrootd-archive.cr.cnaf.infn.it:1096/ ls /
xrdcp root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt ./test260403

Thanks,
Andrea

OK here are results both with and without XrdSecDEBUG=1
so it is trying the ztn protocol like it should but it is failing.

<> xrdfs root://xrootd-archive.cr.cnaf.infn.it:1096/ ls /
xrdcp root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt ./test260403
[ERROR] Server responded with an error: [3010] Unable to locate /; permission denied

[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3010] Unable to open /dune/testpro/bb/7f/awt-download-2023-03-07-01.txt; permission denied (source)

<> xrdfs root://xrootd-archive.cr.cnaf.infn.it:1096/ ls /
[ERROR] Server responded with an error: [3010] Unable to locate /; permission denied

<> xrdcp root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt ./test260403
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3010] Unable to open /dune/testpro/bb/7f/awt-download-2023-03-07-01.txt; permission denied (source)

<> export XrdSecDEBUG=1
<> xrdfs root://xrootd-archive.cr.cnaf.infn.it:1096/ ls /dune
sec_Client: protocol request for host xrootd-archive.cr.cnaf.infn.it token='&P=gsi,v:10600,c:ssl,ca:82977a7e.0|5a04724c.0&P=ztn,0:4096:'
sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
Secgsi -------------------------------------------------------------------
Secgsi Mode: client
Secgsi Debug: 1
Secgsi CA dir: /etc/grid-security/certificates/
Secgsi CA verification level: verifyss
Secgsi CRL dir: /etc/grid-security/certificates/
Secgsi CRL extension: .r0
Secgsi CRL check level: try
Secgsi CRL refresh time: 86400
Secgsi Certificate: /nashome/t/timm/.globus/usercert.pem
Secgsi Key: /nashome/t/timm/.globus/userkey.pem
Secgsi Proxy file: /tmp/x509up_u2904
Secgsi Proxy validity: 12:00
Secgsi Proxy dep length: 0
Secgsi Proxy bits: 2048
Secgsi Proxy sign option: 1
Secgsi Proxy delegation option: 0
Secgsi Pure Cert/Key authentication allowed
Secgsi Allowed server names: [*/]<target host name>[/*]
Secgsi Crypto modules: ssl
Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
Secgsi MDigests: sha256
Secgsi Trusting DNS for hostname checking
Secgsi -------------------------------------------------------------------
sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:82977a7e.0|5a04724c.0'
260403 09:23:12 1426839 cryptossl_X509::CertType: certificate has 8 extensions
260403 09:23:12 1426839 secgsi_VerifyCA: Warning: CA certificate not self-signed and integrity not checked: assuming OK (82977a7e.0)
260403 09:23:12 1426839 cryptossl_X509::CertType: certificate has 8 extensions
260403 09:23:12 1426839 secgsi_QueryProxy: problems initializing proxy via external shell
sec_Client: protocol request for host xrootd-archive.cr.cnaf.infn.it token='&P=ztn,0:4096:'
sec_PM: Loaded ztn protocol object from libXrdSecztn.so
sec_PM: Using ztn protocol, args='0:4096:'
[ERROR] Server responded with an error: [3010] Unable to locate /dune; permission denied

<> xrdcp root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt ./test260403
sec_Client: protocol request for host xrootd-archive.cr.cnaf.infn.it token='&P=gsi,v:10600,c:ssl,ca:82977a7e.0|5a04724c.0&P=ztn,0:4096:'
sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
Secgsi -------------------------------------------------------------------
Secgsi Mode: client
Secgsi Debug: 1
Secgsi CA dir: /etc/grid-security/certificates/
Secgsi CA verification level: verifyss
Secgsi CRL dir: /etc/grid-security/certificates/
Secgsi CRL extension: .r0
Secgsi CRL check level: try
Secgsi CRL refresh time: 86400
Secgsi Certificate: /nashome/t/timm/.globus/usercert.pem
Secgsi Key: /nashome/t/timm/.globus/userkey.pem
Secgsi Proxy file: /tmp/x509up_u2904
Secgsi Proxy validity: 12:00
Secgsi Proxy dep length: 0
Secgsi Proxy bits: 2048
Secgsi Proxy sign option: 1
Secgsi Proxy delegation option: 0
Secgsi Pure Cert/Key authentication allowed
Secgsi Allowed server names: [*/]<target host name>[/*]
Secgsi Crypto modules: ssl
Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
Secgsi MDigests: sha256
Secgsi Trusting DNS for hostname checking
Secgsi -------------------------------------------------------------------
sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:82977a7e.0|5a04724c.0'
260403 09:23:52 1427478 cryptossl_X509::CertType: certificate has 8 extensions
260403 09:23:52 1427478 secgsi_VerifyCA: Warning: CA certificate not self-signed and integrity not checked: assuming OK (82977a7e.0)
260403 09:23:52 1427478 cryptossl_X509::CertType: certificate has 8 extensions
260403 09:23:52 1427478 secgsi_QueryProxy: problems initializing proxy via external shell
sec_Client: protocol request for host xrootd-archive.cr.cnaf.infn.it token='&P=ztn,0:4096:'
sec_PM: Loaded ztn protocol object from libXrdSecztn.so
sec_PM: Using ztn protocol, args='0:4096:'
[0B/0B][100%][==================================================][0B/s]
Run:

Hi,

thanks a lot for the feedback.
Could you please try the following test, forcing the XRootD client to use the token only (without X.509)?

$ export BEARER_TOKEN=$(oidc-token <dune>)
$ export XrdSecPROTOCOL=ztn
$ unset X509_USER_PROXY
$
$ export XrdSecDEBUG=1
$ xrdfs root://xrootd-archive.cr.cnaf.infn.it:1096/ ls /dune
$ xrdcp root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt ./test_token

Please send us the full output.

Can you also retry with a token that has only the "storage.read:/" scope?

In particular, we would like to confirm that only the ztn protocol is used (and no gsi appears in the debug output).

Thanks,
Andrea

Dear site admins,I would like to report that the site INFN-T1, especially INFN-T1_DATATAPE has been failing as a source for transfers and staging from this site endpoint, with ~4.13K transfer failures over the past 6h.
The most common errors are:

STAGING [22] [Tape REST API] Stage call failed: HTTP 502 : Unexpected server error: 502 : <html> <head><title>502 Bad Gateway</PATH <body> <center><h1>502 Bad Gateway</PATH <hr><center>nginx/PATH </body> </html>

TRANSFER ERROR: Copy failed (3rd pull). Last attempt: copy HTTP 500 : Unexpected server error: 500

The site has also shown a high rate of deletion failure (only ~50% deletion efficiency over the past 6h)

Link: https://monit-grafana.cern.ch/d/FtSFfwdmk/ddm-transfers?orgId=17&var-binning=%24__auto_interval_binning&var-groupby=src_experiment_site&var-activity=Analysis+Input&var-activity=Analysis+Output&var-activity=Data+Carousel+Analysis&var-activity=Data+Carousel+Production&var-activity=Data+Challenge&var-activity=Data+Consolidation&var-activity=Data+Rebalancing&var-activity=Deletion&var-activity=Express&var-activity=Functional+Test&var-activity=Production+Input&var-activity=Production+Output&var-activity=Recovery&var-activity=Staging&var-activity=T0+Export&var-activity=T0+Tape&var-activity=T0+Tape+Derived&var-activity=T0+Tape+RAW&var-activity=User+Subscriptions&var-activity=default&var-activity=T0+Recall&var-src_tier=0&var-src_tier=1&var-src_tier=2&var-src_country=All&var-src_cloud=All&var-src_site=INFN-T1&var-src_endpoint=All&var-src_token=All&var-columns=src_experiment_site&var-dst_tier=0&var-dst_tier=1&var-dst_tier=2&var-dst_country=All&var-dst_cloud=All&var-dst_site=All&var-dst_endpoint=All&var-dst_token=All&var-rows=dst_experiment_site&var-measurement=ddm_transfer&var-retention_policy=raw&var-include=&var-exclude=TEST%7CPPS%7CGRIDFTP%7CLAKE%7CAWS&var-exclude_es=All&var-include_es_dst=All&var-include_es_src=All&var-activity_disabled=Analysis+Input&var-activity_disabled=Data+Consolidation&var-activity_disabled=Deletion&var-activity_disabled=Functional+Test&var-activity_disabled=Production+Input&var-activity_disabled=Production+Output&var-activity_disabled=User+Subscriptions&var-protocol=All&var-remote_access=All&from=1773257283289&to=1773278883289

Example:

03/12/2026, 12:20:16 AM

Data Carousel Production

data17_13TeV

AOD.28874557._000032.pool.root.1

STAGING [22] [Tape REST API] Stage call failed: HTTP 502 : Unexpected server error: 502 : <html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx/1.28.2</center> </body> </html> )

transfer-failed

INFN-T1_DATATAPE

CERN-PROD_DATADISK

12.9 hours

1.23 GB

https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/01885592-1d3d-11f1-bd7f-fa163ea7ee69

davs://xfer-tape-atlas.cr.cnaf.infn.it:8443/atlas-tape/atlasdatatape/data17_13TeV/AOD/r13575_p5087/data17_13TeV.00338377.physics_BphysLS.merge.AOD.r13575_p5087_tid28874557_00/AOD.28874557._000032.pool.root.1?copy_mode=pull

davs://eosatlas.cern.ch:443/eos/atlas/atlasdatadisk/rucio/data17_13TeV/ea/3a/AOD.28874557._000032.pool.root.1

davs

1228974757

-1475439524-1773274816000

1773276618

1773228166000

data17_13TeV.00338377.physics_BphysLS.merge.AOD.r13575_p5087_tid28874557_00

data17_13TeV

AOD

UNKNOWN

true

UNKNOWN

447465609

1773276611000

Please have a look.

Khanh (ADCoS shift).

Thank you for your patience. Due to a temporary reduction in help
desk staffing over the next two months, you may experience longer than
usual response and resolution times. Please be assured that we are
working diligently to address your request as quickly as possible.

Hi
Khanh,
could you share an FTS log file?
Unfortunately, I cnanot open the link https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/01885592-1d3d-11f1-bd7f-fa163ea7ee69
and I find pretty hard to debug anything using Monit.
Thanks for you help,
lucia

Hi again,
concerning the specific transfer you mentioned, I checked the log files of our services.

The recall process succeeded at 03:17:34 today:

03/12/26 03:17:34 tsm-hsm-7 yamssProcessRecall[2540341]: Recall of file /storage/gpfs_tsm_atlas/atlas/atlasdatatape/data17_13TeV/AOD/r13575_p5087/data17_13TeV.00338377.physics_BphysLS.merge.AOD.r13575_p5087_tid28874557_00/AOD.28874557._000032.pool.root.1 succeded
the file is on buffer (it still is), and xfer-atlas has correctly performed a GET at 04:09:18 today:

2026-03-12T04:09:18+01:00 [a4f382aa3d998cb9da7448f727ddb6a5] 2001:1458:301:27::100:75 - "-" "GET /atlas-tape/atlasdatatape/data17_13TeV/AOD/r13575_p5087/data17_13TeV.00338377.physics_BphysLS.merge.AOD.r13575_p5087_tid28874557_00/AOD.28874557._000032.pool.root.1?copy_mode=pull HTTP/1.1" "xrootd-tpc/5.9.1" 200 1228974757 6.849
where you see all the 1228974757 bytes seem correctly transfered...

I don't see a problem but please help me,
lucia

Hi,

Here is an example of a transfer failure as source:

12/03/2026, 20:24:44

Data Carousel Production

data18_13TeV

AOD.28681953._000369.pool.root.1

TRANSFER
ERROR: Copy failed (3rd pull). Last attempt: Transfer failure: socket
timeout on GET (received 491135 B (480 KiB) of data; 0 B pending): Read
timed out

transfer-failed

INFN-T1_DATATAPE

PIC_DATADISK

14.3 mins

3.76 GB

https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/f2bb19a6-1e49-11f1-a4fe-fa163e2cda95

davs://xfer-tape-atlas.cr.cnaf.infn.it:8443/atlas-tape/atlasdatatape/data18_13TeV/AOD/r13313_p4910/data18_13TeV.00359823.physics_BphysLS.merge.AOD.r13313_p4910_tid28681953_00/AOD.28681953._000369.pool.root.1?copy_mode=pull

davs://webdav-at1.pic.es:8446/atlasdatadisk/rucio/data18_13TeV/a5/e5/AOD.28681953._000369.pool.root.1

davs

3759350859

2058218065-1773347084000

1773347101

1773343767000

data18_13TeV.00359823.physics_BphysLS.merge.AOD.r13313_p4910_tid28681953_00

data18_13TeV

AOD

UNKNOWN

true

UNKNOWN

445520297

177334708

https://monit-grafana.cern.ch/d/FtSFfwdmk/ddm-transfers?orgId=17&var-binning=%24__auto_interval_binning&var-groupby=src_experiment_site&var-activity=Analysis+Input&var-activity=Analysis+Output&var-activity=Data+Carousel+Analysis&var-activity=Data+Carousel+Production&var-activity=Data+Challenge&var-activity=Data+Consolidation&var-activity=Data+Rebalancing&var-activity=Deletion&var-activity=Express&var-activity=Functional+Test&var-activity=Production+Input&var-activity=Production+Output&var-activity=Recovery&var-activity=Staging&var-activity=T0+Export&var-activity=T0+Recall&var-activity=T0+Tape&var-activity=T0+Tape+Derived&var-activity=T0+Tape+RAW&var-activity=User+Subscriptions&var-activity=default&var-src_tier=0&var-src_tier=1&var-src_tier=2&var-src_country=All&var-src_cloud=All&var-src_site=INFN-T1&var-src_endpoint=All&var-src_token=All&var-columns=src_experiment_site&var-dst_tier=0&var-dst_tier=1&var-dst_tier=2&var-dst_country=All&var-dst_cloud=All&var-dst_site=All&var-dst_endpoint=All&var-dst_token=All&var-rows=dst_experiment_site&var-measurement=ddm_transfer&var-retention_policy=raw&var-include=&var-exclude=TEST%7CPPS%7CGRIDFTP%7CLAKE%7CAWS&var-exclude_es=All&var-include_es_dst=All&var-include_es_src=All&var-activity_disabled=Analysis+Input&var-activity_disabled=Data+Consolidation&var-activity_disabled=Deletion&var-activity_disabled=Functional+Test&var-activity_disabled=Production+Input&var-activity_disabled=Production+Output&var-activity_disabled=Staging&var-activity_disabled=User+Subscriptions&var-protocol=All&var-remote_access=All&var-enr_filters=data.purged_reason%7C%3D%7CTRANSFER+ERROR%3A+Copy+failed+%283rd+pull%29.+Last+attempt%3A+Transfer+failure%3A+socket+timeout+on+GET+%3A+Read+timed+out&from=1773336368508&to=1773347168508

Best regards,

Gabriela (ADCOS shifter)

Hi Luca,
TLS communication hangs for some time / sometimes even timeout, e.g. following transfer gets stuck 3 times for quite some time (40s, 22s, 226s) and there is no packet loss. Are (all?) your servers somehow overloaded? Still it is weird that transfers gets stuck even during TLS handshake
$ curl -s -v --capath /etc/grid-security/certificates -L -I -H "Authorization: Bearer $TSRC" "$SRC" 2>&1 | timestamp
Mar 13 01:16:42.011 0.007 0.007 * Trying 2001:760:4205:128::129:204:8443...
Mar 13 01:16:42.039 0.034 0.027 * Connected to xfer-atlas.cr.cnaf.infn.it (2001:760:4205:128::129:204) port 8443 (#0)
Mar 13 01:16:42.040 0.036 0.002 * ALPN, offering h2
Mar 13 01:16:42.040 0.036 0.000 * ALPN, offering http/1.1
Mar 13 01:16:42.048 0.043 0.008 * CAfile: /etc/pki/tls/certs/ca-bundle.crt
Mar 13 01:16:42.048 0.043 0.000 * CApath: /etc/grid-security/certificates
Mar 13 01:16:42.048 0.043 0.000 * TLSv1.0 (OUT), TLS header, Certificate Status (22):
Mar 13 01:16:42.048 0.043 0.000 } [5 bytes data]
Mar 13 01:16:42.048 0.043 0.000 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
Mar 13 01:16:42.048 0.043 0.000 } [512 bytes data]
Mar 13 01:17:22.295 40.290 40.247 * TLSv1.2 (IN), TLS header, Certificate Status (22):
Mar 13 01:17:22.295 40.291 0.001 { [5 bytes data]
Mar 13 01:17:22.295 40.291 0.000 * TLSv1.3 (IN), TLS handshake, Server hello (2):
Mar 13 01:17:22.295 40.291 0.000 { [122 bytes data]
Mar 13 01:17:22.295 40.291 0.000 * TLSv1.2 (IN), TLS header, Finished (20):
Mar 13 01:17:22.295 40.291 0.000 { [5 bytes data]
Mar 13 01:17:22.295 40.291 0.000 * TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
Mar 13 01:17:22.295 40.291 0.000 { [1 bytes data]
Mar 13 01:17:22.295 40.291 0.000 * TLSv1.2 (IN), TLS header, Unknown (23):
Mar 13 01:17:22.295 40.291 0.000 { [5 bytes data]
Mar 13 01:17:22.295 40.291 0.000 * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
Mar 13 01:17:22.295 40.291 0.000 { [25 bytes data]
Mar 13 01:17:22.295 40.291 0.000 * TLSv1.2 (IN), TLS header, Unknown (23):
Mar 13 01:17:22.295 40.291 0.000 { [5 bytes data]
Mar 13 01:17:22.295 40.291 0.000 * TLSv1.2 (IN), TLS header, Unknown (23):
Mar 13 01:17:22.295 40.291 0.000 { [5 bytes data]
Mar 13 01:17:22.295 40.291 0.000 * TLSv1.3 (IN), TLS handshake, Request CERT (13):
Mar 13 01:17:22.295 40.291 0.000 { [20389 bytes data]
Mar 13 01:17:22.297 40.292 0.001 * TLSv1.2 (IN), TLS header, Unknown (23):
Mar 13 01:17:22.297 40.292 0.000 { [5 bytes data]
Mar 13 01:17:22.297 40.292 0.000 * TLSv1.3 (IN), TLS handshake, Certificate (11):
Mar 13 01:17:22.297 40.292 0.000 { [2104 bytes data]
Mar 13 01:17:22.297 40.292 0.000 * TLSv1.2 (IN), TLS header, Unknown (23):
Mar 13 01:17:22.297 40.292 0.000 { [5 bytes data]
Mar 13 01:17:22.297 40.292 0.000 * TLSv1.3 (IN), TLS handshake, CERT verify (15):
Mar 13 01:17:22.297 40.292 0.000 { [392 bytes data]
Mar 13 01:17:22.297 40.292 0.000 * TLSv1.2 (IN), TLS header, Unknown (23):
Mar 13 01:17:22.297 40.292 0.000 { [5 bytes data]
Mar 13 01:17:22.297 40.292 0.000 * TLSv1.3 (IN), TLS handshake, Finished (20):
Mar 13 01:17:22.297 40.292 0.000 { [52 bytes data]
Mar 13 01:17:22.297 40.292 0.000 * TLSv1.2 (OUT), TLS header, Finished (20):
Mar 13 01:17:22.297 40.292 0.000 } [5 bytes data]
Mar 13 01:17:22.297 40.292 0.000 * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
Mar 13 01:17:22.297 40.292 0.000 } [1 bytes data]
Mar 13 01:17:22.297 40.292 0.000 * TLSv1.2 (OUT), TLS header, Unknown (23):
Mar 13 01:17:22.297 40.293 0.000 } [5 bytes data]
Mar 13 01:17:22.297 40.293 0.000 * TLSv1.3 (OUT), TLS handshake, Certificate (11):
Mar 13 01:17:22.297 40.293 0.000 } [8 bytes data]
Mar 13 01:17:22.297 40.293 0.000 * TLSv1.2 (OUT), TLS header, Unknown (23):
Mar 13 01:17:22.297 40.293 0.000 } [5 bytes data]
Mar 13 01:17:22.297 40.293 0.000 * TLSv1.3 (OUT), TLS handshake, Finished (20):
Mar 13 01:17:22.297 40.293 0.000 } [52 bytes data]
Mar 13 01:17:22.297 40.293 0.000 * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
Mar 13 01:17:22.297 40.293 0.000 * ALPN, server accepted to use http/1.1
Mar 13 01:17:22.297 40.293 0.000 * Server certificate:
Mar 13 01:17:22.297 40.293 0.000 * subject: DC=org; DC=terena; DC=tcs; C=IT; L=Roma; O=Istituto Nazionale di Fisica Nucleare; CN=dm-12-14-07.cr.cnaf.infn.it
Mar 13 01:17:22.297 40.293 0.000 * start date: Nov 27 10:03:20 2025 GMT
Mar 13 01:17:22.297 40.293 0.000 * expire date: Nov 27 10:03:20 2026 GMT
Mar 13 01:17:22.297 40.293 0.000 * subjectAltName: host "xfer-atlas.cr.cnaf.infn.it" matched cert's "xfer-atlas.cr.cnaf.infn.it"
Mar 13 01:17:22.297 40.293 0.000 * issuer: C=GR; O=Hellenic Academic and Research Institutions CA; CN=GEANT TLS RSA 1
Mar 13 01:17:22.297 40.293 0.000 * SSL certificate verify ok.
Mar 13 01:17:22.297 40.293 0.000 * TLSv1.2 (OUT), TLS header, Unknown (23):
Mar 13 01:17:22.297 40.293 0.000 } [5 bytes data]
Mar 13 01:17:22.297 40.293 0.000 > HEAD /webdav/atlas/atlasdatadisk/rucio/mc16_13TeV/af/05/HITS.48953936._022474.pool.root.1 HTTP/1.1
Mar 13 01:17:22.297 40.29

Hi Petr,
thanks for debugging the issue.
Servers are not overloaded, nor is the underlying filesystem. We have evidence that some nginx workers get stucked for some requests, which was not expected.
The StoRM developers are looking into this.
Cheers,
lucia

Hi Petr and everybody,
we've been looking into this issue together with StoRM developers, who identified a possible limitation in the current Nginx configuration.
Hence, we modified the configuration in 4 out of 5 endpoints in the xfer-atlas.cr.cnaf.infn.it alias, leaving only one unchanged to further debug.
"Unfortunately", traffic is now very low, so we do not see any issue at the moment, but we are waiting for you to "push the button" :-)
Thanks for understanding,
lucia

Hi,
from what I can see, the traffic is rather stable since this ticket was submitted:

https://monit-grafana.cern.ch/d/FtSFfwdmk/ddm-transfers?orgId=17&theme=dark&var-binning=1h&var-groupby=src_experiment_site&var-activity=Analysis+Input&var-activity=Analysis+Output&var-activity=Data+Carousel+Analysis&var-activity=Data+Carousel+Production&var-activity=Data+Challenge&var-activity=Data+Consolidation&var-activity=Data+Rebalancing&var-activity=Functional+Test&var-activity=Express&var-activity=Production+Input&var-activity=Production+Output&var-activity=Recovery&var-activity=SFO+to+EOS+export&var-activity=Staging&var-activity=T0+Export&var-activity=T0+Tape&var-activity=T0+Recall&var-activity=T0+Tape+Derived&var-activity=T0+Tape+RAW&var-activity=User+Subscriptions&var-activity=default&var-src_tier=0&var-src_tier=1&var-src_tier=2&var-src_country=All&var-src_cloud=All&var-src_site=INFN-T1&var-src_endpoint=All&var-src_token=All&var-columns=src_cloud&var-dst_tier=0&var-dst_tier=1&var-dst_tier=2&var-dst_country=All&var-dst_cloud=All&var-dst_site=All&var-dst_endpoint=All&var-dst_token=All&var-rows=dst_cloud&var-measurement=ddm_transfer&var-retention_policy=raw&var-include=&var-exclude=TEST%7CPPS%7CGRIDFTP%7CLAKE%7CAWS&var-exclude_es=All&var-include_es_dst=All&var-include_es_src=All&var-activity_disabled=Analysis+Input&var-activity_disabled=Data+Consolidation&var-activity_disabled=Deletion&var-activity_disabled=Express&var-activity_disabled=Functional+Test&var-activity_disabled=Production+Input&var-activity_disabled=Production+Output&var-activity_disabled=Staging&var-activity_disabled=User+Subscriptions&var-protocol=All&var-remote_access=All&from=1773231189372&to=1773835989372&viewPanel=121

and these failures seems to be gone for several days:

https://monit-grafana.cern.ch/d/FtSFfwdmk/ddm-transfers?orgId=17&theme=dark&var-binning=1h&var-groupby=src_experiment_site&var-activity=Analysis+Input&var-activity=Analysis+Output&var-activity=Data+Carousel+Analysis&var-activity=Data+Carousel+Production&var-activity=Data+Challenge&var-activity=Data+Consolidation&var-activity=Data+Rebalancing&var-activity=Functional+Test&var-activity=Express&var-activity=Production+Input&var-activity=Production+Output&var-activity=Recovery&var-activity=SFO+to+EOS+export&var-activity=Staging&var-activity=T0+Export&var-activity=T0+Tape&var-activity=T0+Recall&var-activity=T0+Tape+Derived&var-activity=T0+Tape+RAW&var-activity=User+Subscriptions&var-activity=default&var-src_tier=0&var-src_tier=1&var-src_tier=2&var-src_country=All&var-src_cloud=All&var-src_site=INFN-T1&var-src_endpoint=All&var-src_token=All&var-columns=src_cloud&var-dst_tier=0&var-dst_tier=1&var-dst_tier=2&var-dst_country=All&var-dst_cloud=All&var-dst_site=All&var-dst_endpoint=All&var-dst_token=All&var-rows=dst_cloud&var-measurement=ddm_transfer&var-retention_policy=raw&var-include=&var-exclude=TEST%7CPPS%7CGRIDFTP%7CLAKE%7CAWS&var-exclude_es=All&var-include_es_dst=All&var-include_es_src=All&var-activity_disabled=Analysis+Input&var-activity_disabled=Data+Consolidation&var-activity_disabled=Deletion&var-activity_disabled=Express&var-activity_disabled=Functional+Test&var-activity_disabled=Production+Input&var-activity_disabled=Production+Output&var-activity_disabled=Staging&var-activity_disabled=User+Subscriptions&var-protocol=All&var-remote_access=All&from=1773231234394&to=1773836034394&var-enr_filters=data.purged_reason%7C%3D%7E%7C.*Unexpected.*&viewPanel=122

So, the problem might be fixed. Is there any investigation ongoing or can this ticket be closed?

Hi,

the investigation is still ongoing on our side and unfortunately we do not believe that the new configuration solved the issues reported here.

Are you saying that ATLAS is not observing low efficiency even with a traffic similar to the one ATLAS generated last week, when this ticket was opened?
Thanks for checking,
lucia

Yes, according to that plot I linked, the failures stopped on 13th of March.

Hi,
we enabled asyncronous I/O in all the endpoints given our analysis suggests this improves the handling of GET requests.
It would be great if you could increase the timeout for DELETE requests to 30 seconds, in order to avoid false negatives.
Then, ok for us to close this ticket.
Thanks for your help,
lucia

Transfers and deletions continue to be fine. Let me add the ddmops for your last question/request.

So, it seems the problems have been fixed but I wonder if something related to the addition of ddmops is still pending to proceed with closing the ticket.

Hi,
yes it is... "It would be great if you could increase the timeout for DELETE requests to 30 seconds, in order to avoid false negatives."
Cheers,
lucia

Ciao Lucia, all,

False negatives are less of an issue, true positives taking more than 10 seconds are. When successful, deletions typically complete in less than half a second. [1] In the light of this, I am not sure that increasing the timeout would allow the failures to turn into successes: we had 30 seconds sometime in the past and - with changes effective and communicated on the 27.01.2026 (Reduce timeout from 20s to 10s) - the consensus was that if a namespace operation is doomed to fail anyway, we rather fail early and reattempt later, possibly with a backoff strategy.

That said, I'll raise the subject again within ADC/DDM, but for the purpose of this ticket we can close it.

Cheers,
Fabio for DDM Ops

[1] https://monit-grafana.cern.ch/goto/4ntRrycDR?orgId=17

The point is that deletion requests go into a queue, like all requests. It really doesn't matter if the request then takes 1 ms to complete. This is independent of success or failure.

The deletions issue appears to be resolved. The efficiency for millions of files over the last days is very close to 100%.
Transfers from INFN-T1 are also very close to 100% efficient over the last 7 days.

There are periods of high staging error rates, but these are dominated by timeouts rather than the "unexpected error" first report above.

https://monit-grafana.cern.ch/d/FtSFfwdmk/ddm-transfers?orgId=17&var-binning=%24__auto_interval_binning&var-groupby=src_experiment_site&var-activity=Analysis+Input&var-activity=Analysis+Output&var-activity=Data+Carousel+Analysis&var-activity=Data+Carousel+Production&var-activity=Data+Challenge&var-activity=Data+Consolidation&var-activity=Data+Rebalancing&var-activity=Deletion&var-activity=Express&var-activity=Functional+Test&var-activity=Production+Input&var-activity=Production+Output&var-activity=Recovery&var-activity=Staging&var-activity=T0+Export&var-activity=T0+Recall&var-activity=T0+Tape&var-activity=T0+Tape+Derived&var-activity=T0+Tape+RAW&var-activity=User+Subscriptions&var-activity=default&var-src_tier=0&var-src_tier=1&var-src_tier=2&var-src_country=All&var-src_cloud=All&var-src_site=INFN-T1&var-src_endpoint=All&var-src_token=All&var-columns=src_experiment_site&var-dst_tier=0&var-dst_tier=1&var-dst_tier=2&var-dst_country=All&var-dst_cloud=All&var-dst_site=All&var-dst_endpoint=All&var-dst_token=All&var-rows=dst_experiment_site&var-measurement=ddm_transfer&var-retention_policy=raw&var-include=&var-exclude=TEST%7CPPS%7CGRIDFTP%7CLAKE%7CAWS&var-exclude_es=All&var-include_es_dst=All&var-include_es_src=All&var-activity_disabled=Analysis+Input&var-activity_disabled=Data+Consolidation&var-activity_disabled=Deletion&var-activity_disabled=Functional+Test&var-activity_disabled=Production+Input&var-activity_disabled=Production+Output&var-activity_disabled=User+Subscriptions&var-protocol=All&var-remote_access=All&from=1774859061433&to=1775031861434&viewPanel=155

An FTS log with a bunch of these failures (
STAGING [110] [Tape REST API] Stage pooling call failed: timeout of 300s ):
https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/cab3d0a2-2d02-11f1-9000-fa163e2cda95

Is there anything to do about these timeouts?

Hi Alexander,

we are looking into the problem with the StoRM developers to improve the situation as much as possible.
We’ll let you know as soon as we have something new.

Andrea

Dear site admins,

Following up from < https://its.cern.ch/jira/browse/ATLDDMOPS-5797 | Enable archive monitoring across all tape storages > , we notice a high number of failures as destination for the INFN-T1_MCTAPE endpoint (where the archive monitoring feature was enabled, while it wasn't on DATATAPE so far).

For convenience, you can find the related plots and reports in the DDM Transfer dashoard at https://monit-grafana.cern.ch/goto/bfmhAZ5vR?orgId=17 , specifically in the Transfer Plots, Errors, and Details panels.

The most common error (40k+) reads like "ARCHIVING [11] [Tape REST API] File locality reported as UNAVAILABLE".
This leads to repeated transfer attempts and, while investigating a few replicas, I noticed the we can end up with unnecessary duplicate replicas of the same file. For example (note the _1773763209 timestamp suffix):

```
# rucio list-file-replicas --rses INFN-T1_MCTAPE --pfns mc23_13p6TeV:AOD.46297565._002741.pool.root.1
davs://xfer-tape-atlas.cr.cnaf.infn.it:8443/atlas-tape/atlasmctape/mc23_13p6TeV/AOD/e8592_e8586_s4369_s4370_r16083_r15970/mc23_13p6TeV.700781.Sh_2214_Wmunu_maxHTpTV2_CFilterBVeto.merge.AOD.e8592_e8586_s4369_s4370_r16083_r15970_tid46297565_00/AOD.46297565._002741.pool.root.1_1773763209

# curl -X POST --cert
$X509_USER_PROXY --capath /etc/grid-security/certificates -H
"Content-Type: application/json" https://tape-atlas.cr.cnaf.infn.it:8443/api/v1/archiveinfo
--data
'{"paths":["/atlas-tape/atlasmctape/mc23_13p6TeV/AOD/e8592_e8586_s4369_s4370_r16083_r15970/mc23_13p6TeV.700781.Sh_2214_Wmunu_maxHTpTV2_CFilterBVeto.merge.AOD.e8592_e8586_s4369_s4370_r16083_r15970_tid46297565_00/AOD.46297565._002741.pool.root.1"]}'
| jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 508 100 263 100 245 903 841 --:--:-- --:--:-- --:--:-- 1739
[
{

"path":
"/atlas-tape/atlasmctape/mc23_13p6TeV/AOD/e8592_e8586_s4369_s4370_r16083_r15970/mc23_13p6TeV.700781.Sh_2214_Wmunu_maxHTpTV2_CFilterBVeto.merge.AOD.e8592_e8586_s4369_s4370_r16083_r15970_tid46297565_00/AOD.46297565._002741.pool.root.1",
"locality": "TAPE"
}
]

# curl -X POST --cert $X509_USER_PROXY --capath /etc/grid-security/certificates -H "Content-Type: application/json" https://tape-atlas.cr.cnaf.infn.it:8443/api/v1/archiveinfo --data '{"paths":["/atlas-tape/atlasmctape/mc23_13p6TeV/AOD/e8592_e8586_s4369_s4370_r16083_r15970/mc23_13p6TeV.700781.Sh_2214_Wmunu_maxHTpTV2_CFilterBVeto.merge.AOD.e8592_e8586_s4369_s4370_r16083_r15970_tid46297565_00/AOD.46297565._002741.pool.root.1_1773749267"]}' | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 530 100 274 100 256 913 853 --:--:-- --:--:-- --:--:-- 1766
[
{
"path": "/atlas-tape/atlasmctape/mc23_13p6TeV/AOD/e8592_e8586_s4369_s4370_r16083_r15970/mc23_13p6TeV.700781.Sh_2214_Wmunu_maxHTpTV2_CFilterBVeto.merge.AOD.e8592_e8586_s4369_s4370_r16083_r15970_tid46297565_00/AOD.46297565._002741.pool.root.1_1773749267",
"locality": "TAPE"
}
]
```

For the time being, I have disabled the archive monitoring feature for INFN-T1, but we would like the issue to be addressed and reenabled it again.
Could you please have a look?

Cheers,
Fabio for DDM Ops

Hi Fabio,

thanks a lot for the notification.

We figured out that the problem is strictly related to the proper management of the extended attributes of the files on the tape buffer: there is a brief window of time during which the files can be in the "UNAVAILABLE" status. This is quite normal in our opinion because there is always a migration period during which the files are in an in-transit state.

We are evaluating some possible improvements and fixes, and we'll let you know about them.

However, with respect to the file you mentioned, we tried to look at its history.

The file was written on March 17th at 6:40:

pull third-party transfer completed: DONE. Source: https://se1.farm.particle.cz:443/atlas/atlasdatadisk/rucio/mc23_13p6TeV/b4/bd/AOD.46297565._002741.pool.root.1?copy_mode=pull
, Destination: /atlas-tape/atlasmctape/mc23_13p6TeV/AOD/e8592_e8586_s4369_s4370_r16083_r15970/mc23_13p6TeV.700781.Sh_2214_Wmunu_maxHTpTV2_CFilterBVeto.merge.AOD.e8592_e8586_s4369_s4370_r16083_r15970_tid46297565_00/AOD.46297565._002741.pool.root.1, Bytes transferred: 7243335889, Duration (msec): 207755, Throughput: 33 MB/sec

but then you looked at the locality about 6 hours later, 4 times, with an interval of 10 minutes:

Mar 17 12:36:37
Mar 17 12:46:37
Mar 17 12:56:37
Mar 17 13:06:37

So, some questions naturally arise.

Can you confirm the time between the completed transfer and the first /archiveinfo attempt? Is it possible to configure or extend it up to 24 hours? Or were the observed 6 hours due to a certain load, or queue, on the FTS servers?

How many times can the Tape REST API report an error or the UNAVAILABLE status before FTS declares the transfer as failed and retries?

Can you extend the time between two /archiveinfo requests from 10 minutes to 1 hour?

Have you considered avoiding the retry behaviour where a timestamp is appended to the file name, and instead simply retry transferring the same file?

Thanks a lot,
Andrea

Dear colleagues,

Now that the 2026 data taking has started, we are reviewing the state of the storage
pledges of the LHCb T1 and T2 sites.

Earlier this year, we were informed that (with a few exceptions) the sites would have
the pledged disk and tape resources available to the experiment, and we hope you
can confirm that.

1 - Status of disk pledges

RAL pledges 25003 TB of disk space for 2026. However, the Storage Resource Record (SRR)
advertises only 22092 TB [1]. Do you know what explains that discrepancy?

2 - Status of tape pledges

The site pledges 56281 TB of tape space for 2026. Are these resources available
to LHCb already?

Thanks in advance for your efforts to make these crucial resources available
to us!

best regards, Jan van Eldik / LHCb Compute project lead

[1]
+---------+--------------+-----------+-----------+----------+
| Site | Share | Size | Used | Fraction |
+---------+--------------+-----------+-----------+----------+
| RAL | LHCb-Disk | 22092.0 | 20650.2 | 93.5% |
+---------+--------------+-----------+-----------+----------+

We are in the process of confiming capabilites and timescale for update to the pledges.

Thanks Brian

Hi Brian, do you have an update for us?
Many thanks, Jan

Initial imcrease in pledges with current installed capicities on curent hardware have been calculated and will be deployed. futher increase may come shortly.

Dear site admins,
During the recent file integrity check we found 310 corrupted files on ECHO. The list of files is attached.
Could you help us to investigate the reasons behind the corruption?
Best Regards,
Alex

Will pass onto our storage experts to investigate

Hi,
The files from the list can be grouped as follows:
1. Files uploaded during storage instability periods
a. The most recent one is `buffer/lhcb/MC/2017/SIM/00340649/0009/00340649_00091464_1.sim` -- it was uploaded on the 20th of January this year. First there was an attempt to upload it by a job, which failed, then the file was uploaded to PIC and, after 8(!) unsuccessful attempts it was finally successfully copied to RAL. Job log, as well as FTS logs, are attached. Some FTS transfers reported stuck deletions, and they are probably the reason behind the corruption. On the 20th of January there was indeed an incident with ECHO [1].
b. Other 24 files from this category seem to correspond to 2023 incidents:
bash-5.1$ cat other_incidents.txt
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2011/ALLSTREAMS.DST/00188279/0000/00188279_00000118_5.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2012/B2DHH.STRIP.MDST/00179789/0000/00179789_00000139_1.b2dhh.strip.mdst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2016/ALLSTREAMS.DST/00185904/0000/00185904_00000197_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2016/ALLSTREAMS.DST/00185904/0000/00185904_00000625_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2016/ALLSTREAMS.DST/00185932/0000/00185932_00000034_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2017/ALLSTREAMS.DST/00185750/0000/00185750_00000394_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2017/ALLSTREAMS.DST/00185752/0000/00185752_00000294_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2017/ALLSTREAMS.DST/00185752/0000/00185752_00000421_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2017/ALLSTREAMS.DST/00185902/0000/00185902_00000608_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2017/ALLSTREAMS.DST/00185902/0000/00185902_00000622_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2017/ALLSTREAMS.DST/00185902/0000/00185902_00000630_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2017/ALLSTREAMS.DST/00185906/0000/00185906_00000596_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2017/ALLSTREAMS.DST/00186056/0000/00186056_00000103_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185898/0000/00185898_00000543_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185900/0000/00185900_00000217_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185900/0000/00185900_00000223_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185900/0000/00185900_00000568_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185900/0000/00185900_00000702_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185916/0000/00185916_00000040_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185918/0000/00185918_00000017_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185918/0000/00185918_00000045_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185920/0000/00185920_00000006_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185924/0000/00185924_00000053_7.AllStreams.dst
root://xrootd.echo.stfc.ac.uk//lhcb:prod/lhcb/MC/2018/ALLSTREAMS.DST/00185998/0000/00185998_00000625_7.AllStreams.dst
bash-5.1$Most of them were created between 13th and 16th of May 2023, when there were again some ECHO problems [2].
bash-5.1$ dirac-dms-lfn-metadata other_incidents.txt | grep 'CreationDate'
CreationDate : 2023-05-30 01:58:39
CreationDate : 2023-05-13 08:50:18
CreationDate : 2023-05-13 02:55:46
CreationDate : 2023-05-13 07:42:49
CreationDate : 2023-05-13 10:00:25
CreationDate : 2023-05-13 06:50:29
CreationDate : 2023-05-13 11:28:39
CreationDate : 2023-05-13 07:40:07
CreationDate : 2023-05-13 08:01:21
CreationDate : 2023-05-13 08:37:11
CreationDate : 2023-05-13 06:47:48
CreationDate : 2023-05-13 08:35:24
CreationDate : 2023-05-16 07:11:06
CreationDate : 2023-05-13 06:54:46
CreationDate : 2023-05-13 02:24:12
CreationDate : 2023-05-13 02:26:08
CreationDate : 2023-05-13 06:15:50
CreationDate : 2023-05-13 07:12:13
CreationDate : 2023-05-13 09:52:33
CreationDate : 2023-05-13 09:51:24
CreationDate : 2023-05-13 11:28:52
CreationDate : 2023-05-13 07:32:42
CreationDate : 2023-05-13 11:32:47
CreationDate : 2023-05-15 14:25:00
bash-5.1$The file `/lhcb/MC/2011/ALLSTREAMS.DST/00188279/0000/00188279_00000118_5.AllSt