Pelican 1.0.1

Bug Fixes

• Fixes issue with pelican that prevented PFBs with more than 10k entries
  from being created. (#78)
• fix out of date docker files (#71)

Improvements

• Support for AWS SSE version 4 (#78)
• Fence client is not required if access_format != "guid" (#70)

Pelican export: temporary files expiration

Release Notes

For: uc-cdis/pelican

Notes since tag: 1.0.0

Notes to tag/commit: f662956

Generated: 2023-02-14

New Features

• Pelican export: store the generated PFB files' _expires_at timestamp in
  MDS (set to 14 days after creation) so they can be deleted once expired
  (#69)
• reverting base image change (#63)
• installing ca-cartificates (#63)
• reverting java update (#63)
• new base image for docker files to update lets encrypt certs (#62)

Breaking Changes

• The Pelican export job now requires access to submit data in MDS. In
  cloud-automation setups, run kubectl delete secrets pelicanservice-g3auto, rm Gen3Secrets/g3auto/pelicanservice/config.json
  and gen3 kube-setup-pelicanjob. Then, grant the pelican-export-job
  client access to (resource=/mds_gateway, method=access, service=mds_gateway) in the user.yaml. (#69)

Bug Fixes

• don't attempt to query if id_chunks is False-y for whatever reason (as this
  causes an exception) (#64)
• Fix bug when launching an export job with an empty input in the body (#61)
• Fix bug when launching an export job with a Sower ROOT_NODE index name
  that is not the name of a node in the graph data model (#61)
• adding aligned_reads_index to the extra nodes on VCF file exports to make
  them inline with CRAM files (#58)

Improvements

• Removing Veracode scanning pipeline (#66, #67)
• Reorganize documentation to make it easier to find and read (#61)
• Document how to launch an export or import job (#61)
• Remove unused _get_dfs function (#61)
• Run black (#61)
• Remove custom PR template (#61)

Dependency Updates

• Use poetry<1.2 to avoid a dependency issue between poetry which needs a
  recent importlib-metadata, and pypfb with uses an older one:
  TypeError: entry_points() got an unexpected keyword argument 'group'
  (#68)

Deployment Changes

• The Pelican export job should now be deployed alongside the
  metadata-delete-expired-objects cronjob: add
  "metadata-delete-expired-objects": "quay.io/cdis/metadata-delete-expired-objects:<version>" to the versions
  block of the manifest, run gen3 kube-setup-metadata-delete-expired-objects-cronjob and grant the
  metadata-delete-expired-objects-job client access to
  (resource=/mds_gateway, method=access, service=mds_gateway) and
  (resource=/programs, method=delete, service=fence) in the user.yaml.
  (#69)

Fix unicode encoding issue with export

use latest version of pypfb to fix encoding (#57)

* use latest version of pypfb to fix encoding

* updating poetry lock file

* fix(version): limit version with gt

Co-authored-by: Michael Lukowski <[email protected]>

20.1.7

Release Notes

For: uc-cdis/pelican

Notes since tag: 2021.06

Notes to tag/commit: 20.1.7

Generated: 2021-06-17

Bug Fixes

• bumping pypfb version to fix export problems (#54)

0.6.1: Adopt Poetry dependency mgmt.

New Features

• Adopt Poetry dependency mgmt.

0.5.1: Bugfix: Childless Nodes

Release Notes

For: uc-cdis/pelican

Notes since tag: 0.5.0

Notes to tag/commit: 69aaf00

Generated: 2020-09-25

Bug Fixes

• Fix bug exporting childless nodes: If exporting a cohort with no children, pelican-export was returning an empty PFB, instead of a PFB containing entities on the root_node and its parents. This was caused by pelican delaying writes to the PFB until encountering a child node (L148). If pelican did not encounter a child node, no nodes would be written to the PFB. (#35)

Allow limited export of data files; compatibility with Tube >= 0.4.0

Release Notes

For: uc-cdis/pelican

Notes since tag: 0.4.0

Notes to tag/commit: 74266f4

Generated: 2020-09-08

New Features

• Pelican-export job now allows client to specify the root node of a pfb when
  dispatching a job, allowing clients to export PFBs of entities that are not
  on the $ROOT_NODE environment variable. This allows for exporting of data
  file PFBs, so long as the data files are all on the same root node. (#33)
• The pelican-export algorithm's extra_nodes input can now be configured by
  $EXTRA_NODES environment variable. The old behavior of including
  reference_file and reference_file_index for BDCat exports is still
  there for backwards compatibility. (#33)

Breaking Changes

• Requires tube version>=0.4.0, which changes the root node id to have a
  prepended _ in elastic search indices (#30)

Bug Fixes

• After some analysis, we found out the PFB export (pyPfb encoding) works
  fine with dictionaryutils 3.0.2. (#32)
• Updated root node id to have new format with prepended _ (#30)

Dependency Updates

• Update Pipfile.lock (#31)

use revproxy to make guppy requests

Release Notes

For: uc-cdis/pelican
Notes since tag: 0.3.2
Notes to tag/commit: 0.4.0
Generated: 2020-03-05

Improvements

• use revproxy to send requests to Guppy instead of going out to the internet
  via commons hostname (#27)

0.3.2

Release Notes

For: uc-cdis/pelican

Notes since tag: 0.3.1

Notes to tag/commit: 0.3.2

Generated: 2020-01-27

New Features

• Added .secinclude to include all relevant files needed for veracode scan.
  (#24)
• Added .secexclude to explicitly remove any unneeded files. (#24)
• Added Jenkinsfile.security for the veracode security organization in
  Jenkins to scan this repo regularly. (#24)

0.3.0

Release Notes

For: uc-cdis/pelican

Notes since tag: 0.2.3

Notes to tag/commit: 0.3.0

Generated: 2020-01-16

New Features

• Adding extra nodes related to reference files. (#21 )
  fix(DataSTAGE): Missing CRAI files (#22)

  • feat: limit Spark driver & executor memory

  • feat: Python 3.7 support with Stretch image

  • feat: unbuffered Python output - useful for getting logs in real-time in kubectl logs

  • feat: chaining iterators

  • feat: increase db fetch size

  • feat: f-string formatting

  • feat: dev requirements for responses

  • feat: refactor for GuppyQL quering

  • feat: rollback Spark memory

  Authored-by: Andrew Prokhorenkov [email protected]