APNIC rsync failback and LANIC AS0

[sarasalingam]

In the cloudflare OctoRPKI, we have disabled the rrdp failover "-rrdp.failover=false" but still for few URLs its going for the rsync (For APNIC)

Please find below log for the same –

TYPE rsync_errors gauge

rsync_errors {address="rsync://rpki-repository.nic.ad.jp/ap"} 146
rsync_errors{address="rsync://rpki.apnic.net/repository"} 146

Could you please help us to identify the possible issue? We understand why is the cause for JPNIC but not for APNIC ? Even though rsync fails, the ROA counts are correct for APNIC compared to the public sites since it is downloading via RRDP.

Nov 19 08:34:16 rpki01 bbe10dbee28e[1531]: time="2021-11-18T21:34:16Z" level=info msg="RRDP: Downloading root notification https://rrdp.apnic.net/notification.xml"
Nov 19 08:34:16 rpki01 bbe10dbee28e[1531]: time="2021-11-18T21:34:16Z" level=info msg="RRDP: https://rrdp.apnic.net/notification.xml has 3 deltas to parse (cur: 95753, last: 95750)"
Nov 19 08:37:00 rpki01 bbe10dbee28e[1531]: time="2021-11-18T21:37:00Z" level=info msg="RRDP sync https://rrdp.sub.apnic.net/notification.xml"
Nov 19 08:37:00 rpki01 bbe10dbee28e[1531]: time="2021-11-18T21:37:00Z" level=info msg="RRDP: Downloading root notification https://rrdp.sub.apnic.net/notification.xml"
Nov 19 08:37:00 rpki01 bbe10dbee28e[1531]: time="2021-11-18T21:37:00Z" level=info msg="RRDP: https://rrdp.sub.apnic.net/notification.xml has 0 deltas to parse (cur: 1696, last: 1696)"

Also we have noticed, OctoRPKI is not fetching the data for LANIC-AS0 tal. Please advise why it is working for APNIC AS0 not for LANIC AS0.

100 17736 17714 66 Nov18 ? 1-11:56:46 ./octorpki -tal.root=tals/afrinic.tal,tals/apnic.tal,tals/arin.tal,tals/lacnic.tal,tals/ripe.tal,tals/lacnic-as0.tal -tal.name=AFRINIC,APNIC,ARIN,LACNIC,RIPE,LACNIC-AS0 -output.sign=false -rrdp.failover=false -refresh=600
100 17896 17876 6 Nov18 ? 03:43:42 ./gortr -loglevel debug -refresh 600 -rtr.refresh 600 -slurm /configs/slurm.json -ssh.bind :8282 -ssh.key private_new.pem -ssh.method.password=true -ssh.auth.user rpki -ssh.auth.password rpki -bind :8283 -cache http://octorpk:8081/output.json -verify=false

[ties]

What version of octorpki are you running - especially if using docker (since the public image is not up to date)?

[ties]

And for the LACNIC AS0 tal: Please check the content of the tal file. I had to manually add line breaks.

[sarasalingam]

Hi ties, We are running version 1.2.2. We have noticed version 1.3.0 was running extremely slow in the LAB and pre-production. The behaviour with rsync was same for APNIC. Debashish, Can you please provide further comments the about the Docker image ? regards, Skanda Arasalingam

[sarasalingam]

Hi Ties, The same TAL is working for routinator. We checked the file size, for spaces and any special characters. Not sure, what do you mean by adding line breaks ? RPKI RPKI Trust Anchor (lacnic.net)<https://www.lacnic.net/4984/2/lacnic/rpki-rpki-trust-anchor> ***@***.*** Regards, Skanda Arasalingam

[sarasalingam]

***@***.*** tals]$ cat lacnic-as0.tal https://rrdp.lacnic.net/ta/rta-lacnic-rpki-as0.cer rsync://repository.lacnic.net/rpkias0/lacnic/rta-lacnic-rpki-as0.cer MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAhW5FgZ9Foda5ZpboK99IzhnBG4Gu9t0M bzaqUI7rEH70RKbxpYtBguktrwVX3CaK7BiDtxOEtQv6iikt2DyfLZ14tpwoh/1NBqPilb+PfvNC N75LU9WYv5Fy651bC+N9kO7tAZeWY1NhZCYi3FjFjBRvv7IbUuWx5Us+xoV0g1jVVI5PI69Cbp/j 1a3CutCe92yJ5z9VTJQYXPw32ti0gAAERCepr21y4sO4rJiJtdDGk2+ezFzSgvgitX+/aqaoTpsD HCcSu0ScdsuY+XIQuq0f/Pcg/ClwSmRX2M+7nsbiOHv0GP4VubEW14u9lvu+XdpaPcZVBRldaP9h 5I1f2QIDAQAB ***@***.*** tals]$ From: Skanda Arasalingam Sent: Wednesday, 24 November 2021 8:25 PM To: cloudflare/cfrpki ***@***.***>; cloudflare/cfrpki ***@***.***> Cc: Author ***@***.***>; Prajakta Yuvraj patil ***@***.***>; Debashish Mukherjee ***@***.***> Subject: RE: [cloudflare/cfrpki] APNIC rsync failback and LANIC AS0 (Issue #104) Hi Ties, The same TAL is working for routinator. We checked the file size, for spaces and any special characters. Not sure, what do you mean by adding line breaks ? RPKI RPKI Trust Anchor (lacnic.net)<https://www.lacnic.net/4984/2/lacnic/rpki-rpki-trust-anchor> ***@***.*** Regards, Skanda Arasalingam -email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[ties]

Hi Skanda,

I see the same whitespace that was in my tal file in that file (for example, before "5l1f"). It looks like OctoRPKI supports line breaks but not spaced within the lines.

# in your paste, there are whitespaces within a line:
...
1a3CutCe92yJ5z9VTJQYXPw32ti0gAAERCepr21y4sO4rJiJtdDGk2+ezFzSgvgitX+/aqaoTpsD HCcSu0ScdsuY+XIQuq0f/Pcg/ClwSmRX2M+7nsbiOHv0GP4VubEW14u9lvu+XdpaPcZVBRldaP9h 5I1f2QIDAQAB
# versus
$ cat ./lacnic-as0.tal
https://rrdp.lacnic.net/ta/rta-lacnic-rpki-as0.cer
rsync://repository.lacnic.net/rpkias0/lacnic/rta-lacnic-rpki-as0.cer

MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAhW5FgZ9Foda5ZpboK99IzhnBG4Gu9t0M
bzaqUI7rEH70RKbxpYtBguktrwVX3CaK7BiDtxOEtQv6iikt2DyfLZ14tpwoh/1NBqPilb+PfvNC
N75LU9WYv5Fy651bC+N9kO7tAZeWY1NhZCYi3FjFjBRvv7IbUuWx5Us+xoV0g1jVVI5PI69Cbp/j
1a3CutCe92yJ5z9VTJQYXPw32ti0gAAERCepr21y4sO4rJiJtdDGk2+ezFzSgvgitX+/aqaoTpsD
HCcSu0ScdsuY+XIQuq0f/Pcg/ClwSmRX2M+7nsbiOHv0GP4VubEW14u9lvu+XdpaPcZVBRldaP9h
5I1f2QIDAQAB

If you edit it such as in the attached copy it works for me:
lacnic-as0.tal.zip

[ties]

I spotted this by running with -loglevel debug:

octorpki_1                   | time="2021-11-24T10:12:03Z" level=info msg="Validator started"
octorpki_1                   | time="2021-11-24T10:12:03Z" level=info msg="Serving HTTP on :8081/output.json"
octorpki_1                   | time="2021-11-24T10:12:03Z" level=debug msg="Fetching /tals/lacnic-as0.tal->/tals/lacnic-as0.tal"
octorpki_1                   | time="2021-11-24T10:12:03Z" level=info msg="Still exploring. Revalidating now"
octorpki_1                   | time="2021-11-24T10:12:03Z" level=error msg="file error for certificate: illegal base64 data at input byte 76"
octorpki_1                   | time="2021-11-24T10:12:03Z" level=debug msg="Fetching /tals/lacnic-as0.tal->/tals/lacnic-as0.tal"
octorpki_1                   | time="2021-11-24T10:12:03Z" level=info msg="Stable state. Revalidating in 10m0s"
octorpki_1                   | time="2021-11-24T10:12:03Z" level=error msg="file error for certificate: illegal base64 data at input byte 76"

[sarasalingam]

Hi Skanda, Please find the attached Dockerfile for your reference. Also, regarding the issue that we are currently facing for APNIC in Cloudflare. We have disabled the rrdp.failover please find below for same. command: "-tal.root=tals/afrinic.tal,tals/apnic.tal,tals/arin.tal,tals/lacnic.tal,tals/ripe.tal,tals/apnic-as0.tal,tals/lacnic-as0.tal -tal.name<https://protect2.fireeye.com/v1/url?k=1dcef56c-4255cc69-1dcec7a1-86d2114eab2f-c2272b6693ff66bf&q=1&e=cdd172cd-3250-47b9-99a4-5181e6487606&u=http%3A%2F%2Ftal.name%2F>=AFRINIC,APNIC,ARIN,LACNIC,RIPE,APNIC-AS0,LACNIC-AS0 -output.sign=false -rrdp.failover=false -refresh=10m -loglevel=debug" But, validator is still fetching the rsync for apnic. Please find below and also please find the attached log file (rpki01.meb) – Nov 25 15:10:58 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:10:58Z" level=debug msg="Fetching rsync://rpki.sub.apnic.net/repository/A9114E750000/0/0108398CA988382C2A509BFDB39E146A76CF9DE0.mft->cache/rpki.sub.apnic.net/repository/A9114E750000/0/0108398CA988382C2A509BFDB39E146A76CF9DE0.mft" We have checked the TAL file for APNIC as well the rsync is after the https link. Below is the TAL file that we are currently using. https://rpki.apnic.net/repository/apnic-rpki-root-iana-origin.cer rsync://rpki.apnic.net/repository/apnic-rpki-root-iana-origin.cer MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAx9RWSL61YAAYumEiU8z8 qH2ETVIL01ilxZlzIL9JYSORMN5Cmtf8V2JblIealSqgOTGjvSjEsiV73s67zYQI 7C/iSOb96uf3/s86NqbxDiFQGN8qG7RNcdgVuUlAidl8WxvLNI8VhqbAB5uSg/Mr LeSOvXRja041VptAxIhcGzDMvlAJRwkrYK/Mo8P4E2rSQgwqCgae0ebY1CsJ3Cjf i67C1nw7oXqJJovvXJ4apGmEv8az23OLC6Ki54Ul/E6xk227BFttqFV3YMtKx42H cCcDVZZy01n7JjzvO8ccaXmHIgR7utnqhBRNNq5Xc5ZhbkrUsNtiJmrZzVlgU6Ou 0wIDAQAB Ideally it should not fallback to rsync but it is falling back for APNIC. Could you please advise what could be the possible issue here? Thank you. Kind regards, Prajakta Patil. From: Skanda Arasalingam ***@***.***> Sent: Wednesday, 24 November 2021 8:07 PM To: cloudflare/cfrpki ***@***.***>; cloudflare/cfrpki ***@***.***> Cc: Author ***@***.***>; Debashish Mukherjee ***@***.***>; Prajakta Yuvraj patil ***@***.***> Subject: RE: [cloudflare/cfrpki] APNIC rsync failback and LANIC AS0 (Issue #104) Hi ties, We are running version 1.2.2. We have noticed version 1.3.0 was running extremely slow in the LAB and pre-production. The behaviour with rsync was same for APNIC. Debashish, Can you please provide further comments the about the Docker image ? regards, Skanda Arasalingam From: Ties de Kock ***@***.******@***.***>> Sent: Wednesday, 24 November 2021 6:12 PM To: cloudflare/cfrpki ***@***.******@***.***>> Cc: Skanda Arasalingam ***@***.******@***.***>>; Author ***@***.******@***.***>> Subject: Re: [cloudflare/cfrpki] APNIC rsync failback and LANIC AS0 (Issue #104) [External email] Please be cautious when clicking on any links or attachments. --- What version of octorpki are you running - especially if using docker (since the public image is not up to date)? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#104 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVANCVGBIYHRT4BGVVXT2ELUNSF3NANCNFSM5ITB5BOA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. Nov 25 15:07:39 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:07:39Z" level=debug msg="Fetching rsync://rpki.apnic.net/member_repository/A91DCD09/EAC7EEA0112D11EC8D87057DC4F9AE02/rbmJeP11H5nZqOsq5y08DFUEU6Q.mft->cache/rpki.apnic.net/member_repository/A91DCD09/EAC7EEA0112D11EC8D87057DC4F9AE02/rbmJeP11H5nZqOsq5y08DFUEU6Q.mft" Nov 25 15:07:39 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:07:39Z" level=debug msg="Fetching rsync://rpki.apnic.net/member_repository/A919EAAD/87C3168809D511EAB76C7317C4F9AE02/9TmDt6uDhsaZWHhP-vDmlyYkV6c.mft->cache/rpki.apnic.net/member_repository/A919EAAD/87C3168809D511EAB76C7317C4F9AE02/9TmDt6uDhsaZWHhP-vDmlyYkV6c.mft" Nov 25 15:07:39 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:07:39Z" level=debug msg="Fetching rsync://rpki.apnic.net/member_repository/A91972B6/08243C8C132B11E9AE77AD7BC4F9AE02/leqvwkta9LFytYuwGnAc_hltoZk.mft->cache/rpki.apnic.net/member_repository/A91972B6/08243C8C132B11E9AE77AD7BC4F9AE02/leqvwkta9LFytYuwGnAc_hltoZk.mft" Nov 25 15:07:39 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:07:39Z" level=debug msg="Fetching rsync://rpki.apnic.net/member_repository/A917E678/B897C582A19C11EB92A2DF29C4F9AE02/4kPzvoJVabsWrLN0vJjavmkLF8E.mft->cache/rpki.apnic.net/member_repository/A917E678/B897C582A19C11EB92A2DF29C4F9AE02/4kPzvoJVabsWrLN0vJjavmkLF8E.mft" Nov 25 15:10:57 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:10:57Z" level=debug msg="Fetching rsync://rpki.sub.apnic.net/repository/A9192A980000/4/F67E02BEC4DE61E2EB5953B5DC335E56FFCDCE70.mft->cache/rpki.sub.apnic.net/repository/A9192A980000/4/F67E02BEC4DE61E2EB5953B5DC335E56FFCDCE70.mft" Nov 25 15:10:57 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:10:57Z" level=debug msg="Fetching rsync://rpki.sub.apnic.net/repository/A91905300000/2/84587BF67D2ADA31FC7E0FF93DE2E266D33164C0.mft->cache/rpki.sub.apnic.net/repository/A91905300000/2/84587BF67D2ADA31FC7E0FF93DE2E266D33164C0.mft" Nov 25 15:10:58 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:10:58Z" level=debug msg="Fetching rsync://rpki.sub.apnic.net/repository/A9114E750000/0/0108398CA988382C2A509BFDB39E146A76CF9DE0.mft->cache/rpki.sub.apnic.net/repository/A9114E750000/0/0108398CA988382C2A509BFDB39E146A76CF9DE0.mft" Nov 25 15:12:39 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:12:39Z" level=debug msg="Fetching rsync://rpki.sub.apnic.net/repository/A9192A980000/1/D40581CA9DDACA9E110165B11DD2820DD7F532C0.mft->cache/rpki.sub.apnic.net/repository/A9192A980000/1/D40581CA9DDACA9E110165B11DD2820DD7F532C0.mft" Nov 25 15:12:51 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:12:51Z" level=debug msg="Fetching rsync://rpki.sub.apnic.net/repository/A9192A980000/2/211A048890969FA7D4B6AEF8C020CDA4444EC2E5.mft->cache/rpki.sub.apnic.net/repository/A9192A980000/2/211A048890969FA7D4B6AEF8C020CDA4444EC2E5.mft" Nov 25 15:13:02 rpki01 9cdbcb392fe3[1318]: time="2021-11-25T04:13:02Z" level=debug msg="Fetching rsync://rpki.sub.apnic.net/repository/A9192A980000/3/5EAD10BE7EC295336E4B5680E0D393B677C3649A.mft->cache/rpki.sub.apnic.net/repository/A9192A980000/3/5EAD10BE7EC295336E4B5680E0D393B677C3649A.mft" ARG src_dir="/octorpki" FROM golang:alpine3.14 as builder ARG src_dir ARG LDFLAGS="" RUN apk --update --no-cache add git && \ mkdir -p ${src_dir} WORKDIR ${src_dir} COPY . . RUN go build -ldflags "${LDFLAGS}" cmd/octorpki/octorpki.go FROM alpine:3.14 ARG src_dir RUN apk --update --no-cache add ca-certificates rsync && \ adduser -S -D -H -h / rpki && \ mkdir /cache && chmod 770 /cache && chown rpki:root /cache && \ touch rrdp.json && chown rpki rrdp.json USER rpki COPY --from=builder ${src_dir}/octorpki ${src_dir}/cmd/octorpki/private.pem / COPY --from=builder ${src_dir}/cmd/octorpki/tals /tals VOLUME ["/cache"] ENTRYPOINT ["./octorpki"]

[sarasalingam]

Thanks mate. The LACNIC AS 0 issue is resolved. your advise has resolved the issue for us.

[sarasalingam]

We are not why exactly it is falling back to rsync for APNIC. Please advise.