Skip to content

Sinfo resource management #380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 26 commits into
base: development
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
30c663e
Parse sinfo output to manage resources
antonio-rodriguez-tam Mar 19, 2025
818867c
Add resourceattrtypes to defauls command. Fix parsing of gpu count. P…
antonio-rodriguez-tam Mar 21, 2025
f64bcd5
Use bulk_update_with_history and bulk_create_with_history to generate…
antonio-rodriguez-tam Mar 26, 2025
7d9e4c7
Add current cluster as parent resoruce for compute nodes
antonio-rodriguez-tam Mar 28, 2025
7902069
remove developer configs
claire-peters Apr 17, 2025
61f4c49
prevent creation of Products from Compute Node resources
claire-peters Apr 17, 2025
6af3ce9
change defaults for created resources, set cluster to fasrc
claire-peters Apr 17, 2025
432a628
Include owned resources on user home. Only show owner or access grant…
antonio-rodriguez-tam Mar 20, 2025
f56796c
Only excluded not owned Compute Nodes from resource listview
antonio-rodriguez-tam Mar 26, 2025
a348552
Add listview for retired resources, prevent modifications for retired…
antonio-rodriguez-tam Mar 26, 2025
057d381
Tests
antonio-rodriguez-tam Apr 9, 2025
07ed0aa
Filter our archived resources from user home page managed resources. …
antonio-rodriguez-tam Apr 9, 2025
4be2e3b
add owner property to resource model
claire-peters Apr 18, 2025
e649d44
fix resourceattributetype lookups
claire-peters Apr 18, 2025
c220cab
show owner, link to project if corresponding project exists
claire-peters Apr 18, 2025
5387f36
remove unused imports, revise resource test object creation
claire-peters Apr 18, 2025
0f746a1
Merge pull request #382 from fasrc/include_owned_resources_on_user_home
claire-peters Apr 18, 2025
65f200e
typo
claire-peters Apr 18, 2025
a64e79b
remove RESOURCE_FIXTURES from portal tests
claire-peters Apr 18, 2025
1876bec
fix resource search test
claire-peters Apr 18, 2025
0efc28c
fix resource search test
claire-peters Apr 18, 2025
5356bdf
Remove if statement
antonio-rodriguez-tam Apr 9, 2025
741b906
Don't make products out of Resources that won't be charged anything. …
aaronk Apr 18, 2025
c85b451
change owner identification in resourcedetailview
claire-peters Apr 22, 2025
fd3e401
Merge pull request #384 from fasrc/manage_owner_resource_attribute
claire-peters Apr 22, 2025
51b1f7c
add task for slurm_manage_resources
claire-peters Apr 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions coldfront/config/logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,14 @@
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse'
},
'require_debug_true': {
'()': 'django.utils.log.RequireDebugTrue'
},
},
'formatters': {
'key-events': {
"()": "django.utils.log.ServerFormatter",
Expand All @@ -31,6 +39,12 @@
'console': {
'class': 'logging.StreamHandler',
},
'console_debug': {
'class': 'logging.StreamHandler',
'formatter': 'default',
'level': 'DEBUG',
'filters': ['require_debug_true'],
},
'django-q': {
'class': 'logging.handlers.TimedRotatingFileHandler',
'filename': 'logs/django-q.log',
Expand All @@ -47,6 +61,11 @@
'formatter': 'key-events',
'level': 'INFO',
},
'mail_admins': {
'level': 'ERROR',
'filters': ['require_debug_false'],
'class': 'django.utils.log.AdminEmailHandler',
},
# 'file': {
# 'class': 'logging.FileHandler',
# 'filename': '/tmp/debug.log',
Expand All @@ -66,11 +85,11 @@
'handlers': ['django-q', 'key-events'],
},
'ifx': {
'handlers': ['console', 'key-events'],
'handlers': ['console', 'key-events', 'console_debug', 'mail_admins'],
'level': 'INFO',
},
'ifxbilling': {
'handlers': ['console', 'key-events'],
'handlers': ['console', 'key-events', 'console_debug', 'mail_admins'],
'level': 'INFO',
},
'coldfront': {
Expand Down
1 change: 0 additions & 1 deletion coldfront/core/allocation/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
from django.db.models import Q
from django.utils.html import mark_safe
from django.utils.module_loading import import_string
from django.contrib.auth import get_user_model
from model_utils.models import TimeStampedModel
from simple_history.models import HistoricalRecords

Expand Down
42 changes: 41 additions & 1 deletion coldfront/core/portal/tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@
from coldfront.core.test_helpers import utils
from coldfront.core.test_helpers.factories import setup_models
from coldfront.core.allocation.models import AllocationChangeRequest, AllocationChangeStatusChoice
from coldfront.core.test_helpers.factories import setup_models, ProjectFactory, ResourceFactory, ResourceAttributeTypeFactory, ResourceAttributeFactory
from coldfront.core.resource.models import AttributeType

UTIL_FIXTURES = ['coldfront/core/test_helpers/test_data/test_fixtures/ifx.json']


class PortalViewTest(TestCase):
"""Base class for portal view tests
"""
Expand Down Expand Up @@ -85,3 +86,42 @@ def test_home_page_projects_display(self):
# allocationuser not belonging to project cannot see project
response = utils.login_and_get_page(self.client, self.nonproj_allocationuser, '')
self.assertEqual(response.context['project_list'].count(), 0)

def test_home_page_managed_resources_display(self):
"""check that managed resources display properly on the home page
"""
ProjectFactory(pi=self.pi_user, title="managed_lab")
ProjectFactory(pi=self.admin_user, title="admin_lab")
text_attribute_type = AttributeType.objects.get(name="Text")
managed_resource = ResourceFactory(name="managed_lab", resource_type__name='Compute Node')
managed_resource2 = ResourceFactory(name="managed_lab2", resource_type__name='Compute Node')
admin_resource = ResourceFactory(name="admin_lab", resource_type__name='Compute Node')
owner_resourcer_attr_type = ResourceAttributeTypeFactory(name="Owner", attribute_type=text_attribute_type)
ResourceAttributeFactory(resource_attribute_type=owner_resourcer_attr_type, value="managed_lab",
resource=managed_resource)
ResourceAttributeFactory(resource_attribute_type=owner_resourcer_attr_type, value="managed_lab",
resource=managed_resource2)
ResourceAttributeFactory(resource_attribute_type=owner_resourcer_attr_type, value="admin_lab",
resource=admin_resource)
utils.page_contains_for_user(self, self.pi_user, '', 'Managed Resources')
utils.page_contains_for_user(self, self.admin_user, '', 'Managed Resources')
utils.page_contains_for_user(self, self.pi_user, '', 'managed_lab')
utils.page_contains_for_user(self, self.pi_user, '', 'managed_lab2')
utils.page_does_not_contain_for_user(self, self.pi_user, '', 'admin_lab')
utils.page_contains_for_user(self, self.admin_user, '', 'admin_lab')
utils.page_does_not_contain_for_user(self, self.admin_user, '', 'managed_lab')
utils.page_does_not_contain_for_user(self, self.admin_user, '', 'managed_lab2')

def test_home_page_archive_resources_dont_show(self):
ProjectFactory(pi=self.pi_user, title="managed_lab")
text_attribute_type = AttributeType.objects.get(name="Text")
owner_resourcer_attr_type = ResourceAttributeTypeFactory(name="Owner", attribute_type=text_attribute_type)
archived_resource = ResourceFactory(name="archived_resource", resource_type__name='Compute Node', is_available=False)
archived_resource2 = ResourceFactory(name="archived_resource2", resource_type__name='Compute Node', is_available=False)
active_resource = ResourceFactory(name="active_resource", resource_type__name='Compute Node')
ResourceAttributeFactory(resource_attribute_type=owner_resourcer_attr_type, value="managed_lab", resource=archived_resource)
ResourceAttributeFactory(resource_attribute_type=owner_resourcer_attr_type, value="managed_lab", resource=archived_resource2)
ResourceAttributeFactory(resource_attribute_type=owner_resourcer_attr_type, value="managed_lab", resource=active_resource)
utils.page_contains_for_user(self, self.pi_user, '', 'active_resource')
utils.page_does_not_contain_for_user(self, self.pi_user, '', 'archived_resource')
utils.page_does_not_contain_for_user(self, self.pi_user, '', 'archived_resource2')
12 changes: 7 additions & 5 deletions coldfront/core/portal/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
)
from coldfront.core.project.models import Project
from coldfront.core.publication.models import Publication
from coldfront.core.resource.models import Resource
from coldfront.core.resource.models import Resource, ResourceAttribute
from coldfront.config.env import ENV
from coldfront.core.department.models import Department, DepartmentMember
from coldfront.core.utils.common import import_from_settings
Expand Down Expand Up @@ -82,10 +82,12 @@ def home(request):
department_list = Department.objects.filter(
id__in=user_depts.values_list('organization_id')
)

resource_list = Resource.objects.filter(
allowed_users=request.user)

project_title_list = [project.title for project in project_list]
owned_resources = [attribute.resource.pk for attribute in ResourceAttribute.objects.filter(
resource_attribute_type__name='Owner',
value__in=project_title_list
)]
resource_list = Resource.objects.filter(Q(allowed_users=request.user) | Q(pk__in=owned_resources)).filter(is_available=True).distinct()
context['resource_list'] = resource_list
context['department_list'] = department_list
context['project_list'] = project_list
Expand Down
5 changes: 5 additions & 0 deletions coldfront/core/resource/forms.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ def __init__(self, *args, **kwargs):
self.fields['resource_attribute_type'].queryset = self.fields['resource_attribute_type'].queryset.order_by(Lower('name'))


class ResourceAttributeUpdateForm(forms.Form):
resource_attribute_type_name = forms.CharField(max_length=250, required=False, disabled=True)
value = forms.CharField(max_length=350, required=True)


class ResourceAllocationUpdateForm(forms.Form):
allocation_pk = forms.IntegerField(required=False)
project = forms.CharField(max_length=250, required=False, disabled=True)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def handle(self, *args, **options):
('xdmod_resource', 'Text'),
# ('eula', 'Text'),
# ('OnDemand','Yes/No'),
# ('ServiceEnd', 'Date'),
('ServiceEnd', 'Date'),
# ('ServiceStart', 'Date'),
('slurm_cluster', 'Text'),
('slurm_specs', 'Attribute Expanded Text'),
Expand Down
19 changes: 17 additions & 2 deletions coldfront/core/resource/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from django.db import models
from django.conf import settings
from django.contrib.auth.models import Group
from django.utils.translation import gettext_lazy as _
from django.core.exceptions import ValidationError, ObjectDoesNotExist
from model_utils.models import TimeStampedModel
from simple_history.models import HistoricalRecords
Expand Down Expand Up @@ -194,13 +195,27 @@ def used_percentage(self):
return None

@property
def status(self):
def owner(self):
"""
Returns:
str: the status of the resource
"""
try:
return ResourceAttribute.objects.get(resource=self, resource_attribute_type__name='Owner').value
except ObjectDoesNotExist:
return None

return ResourceAttribute.objects.get(resource=self, resource_attribute_type__attribute='Status').value

@property
def status(self):
"""
Returns:
str: the status of the resource
"""
try:
return ResourceAttribute.objects.get(resource=self, resource_attribute_type__name='Status').value
except ObjectDoesNotExist:
return None

@property
def expiry(self):
Expand Down
63 changes: 63 additions & 0 deletions coldfront/core/resource/templates/resource_archived_list.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{% extends "list_view.html" %}

{% block title %}
Project List
{% endblock %}

{% block page_title%}Archived Resources{% endblock %}

{% block presearch %}
<div class="card mb-3 bg-light">
<div class="card-body">
<div class="float-left">
<a class="btn btn-primary" href="{% url 'resource-list' %}?{{filter_parameters}}" role="button"><i class="fas fa-arrow-left" aria-hidden="true"></i> Back to active resources</a>
</div>
</div>
</div>
{% endblock %}


{% block list_title %}Resource{{count|pluralize}}: {{count}}{% endblock %}

{% block table_contents %}
<thead>
<tr>
<th scope="col" class="text-nowrap">
ID
<a href="?order_by=id&direction=asc&{{filter_parameters}}"><i class="fas fa-sort-up" aria-hidden="true"></i><span class="sr-only">Sort ID asc</span></a>
<a href="?order_by=id&direction=des&{{filter_parameters}}"><i class="fas fa-sort-down" aria-hidden="true"></i><span class="sr-only">Sort ID desc</span></a>
</th>
<th scope="col" class="text-nowrap">
Resource Name
<a href="?order_by=name&direction=asc&{{filter_parameters}}"><i class="fas fa-sort-up" aria-hidden="true"></i><span class="sr-only">Sort Resource Name asc</span></a>
<a href="?order_by=name&direction=des&{{filter_parameters}}"><i class="fas fa-sort-down" aria-hidden="true"></i><span class="sr-only">Sort Resource Name desc</span></a>
</th>
<th scope="col" class="text-nowrap">
Parent Resource
<a href="?order_by=parent_resource&direction=asc&{{filter_parameters}}"><i class="fas fa-sort-up" aria-hidden="true"></i><span class="sr-only">Sort Parent Resource asc</span></a>
<a href="?order_by=parent_resource&direction=des&{{filter_parameters}}"><i class="fas fa-sort-down" aria-hidden="true"></i><span class="sr-only">Sort Parent Resource desc</span></a>
</th>
<th scope="col" class="text-nowrap">
Resource Type
<a href="?order_by=resource_type__name&direction=asc&{{filter_parameters}}"><i class="fas fa-sort-up" aria-hidden="true"></i><span class="sr-only">Sort Resource Type asc</span></a>
<a href="?order_by=resource_type__name&direction=des&{{filter_parameters}}"><i class="fas fa-sort-down" aria-hidden="true"></i><span class="sr-only">Sort Resource Type desc</span></a>
</th>
</tr>
</thead>
<tbody>
{% for resource in item_list %}
<tr>
<td><a href="/resource/{{resource.id}}/">{{ resource.id }}</a></td>
<td>{{ resource }}</td>
<td>{{ resource.parent_resource }}</td>
<td>{{ resource.resource_type.name }}</td>
</tr>
{% endfor %}
</tbody>
{% endblock %}

{% block activelink %}
$("#navbar-project-menu").addClass("active");
$("#navbar-resource").addClass("active");

{% endblock %}
28 changes: 27 additions & 1 deletion coldfront/core/resource/templates/resource_detail.html
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,11 @@

{% block content %}


{% if resource.is_available == False %}
<div class="alert alert-warning" role="alert">
This is a retired resource! You cannot make any changes.
</div>
{% endif %}
<div class="mb-3">
<h2>Resource Detail</h2>
<hr>
Expand All @@ -24,6 +28,11 @@ <h3><i class="fas fa-list" aria-hidden="true"></i> Resource Information</h3>
</div>

<div class="card-body">
{% if request.user.is_superuser and owner != None and owner == '' %}
<div class="alert alert-info" role="alert">
<i class="fas fa-info-circle" aria-hidden="true"></i> No owner has been set for this resource! Change this by editing the Owner value in the Resource Attributes table.
</div>
{% endif %}
{% csrf_token %}
<div class="table-responsive">
<table class="table table-bordered table-sm">
Expand Down Expand Up @@ -73,6 +82,20 @@ <h3><i class="fas fa-list" aria-hidden="true"></i> Resource Information</h3>
{% endif %}
</td>
</tr>
{% if owner != None %}
<tr>
<th scope="row" class="text-nowrap">Owner:</th>
<td>
{% if owner|stringformat:"s" == owner %}
{{ owner }}
{% else %}
<a href="{% url 'project-detail' owner %}">
{{ resource.owner }}
</a>
{% endif %}
</td>
</tr>
{% endif %}
</table>
</div>
</div>
Expand Down Expand Up @@ -100,6 +123,9 @@ <h3 class="d-inline"><i class="fas fa-info-circle" aria-hidden="true"></i> Resou
<a class="btn btn-success" href="{% url 'resource-attribute-add' resource.pk %}" role="button">
<i class="fas fa-plus" aria-hidden="true"></i> Add Resource Attribute
</a>
<a class="btn btn-danger" href="{% url 'resource-attributes-edit' resource.pk %}" role="button">
<i class="fas fa-edit" aria-hidden="true"></i> Edit Resource Attributes
</a>
<a class="btn btn-danger" href="{% url 'resource-attribute-delete' resource.pk %}" role="button">
<i class="fas fa-minus" aria-hidden="true"></i> Delete Resource Attributes
</a>
Expand Down
10 changes: 10 additions & 0 deletions coldfront/core/resource/templates/resource_list.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@
{% block title %}Resource List{% endblock %}

{% block page_title %}Resources{% endblock %}
{% block presearch %}
<div class="card mb-3 bg-light">
<div class="card-body">
<div class="float-right">
<a class="btn btn-primary" href="{% url 'resource-archived-list' %}?{{filter_parameters}}" role="button"><i class="fas fa-archive" aria-hidden="true"></i> View retired resources</a>
</div>
</div>
</div>
{% endblock %}

{% block list_title %}Resource{{count|pluralize}}: {{count}}{% endblock %}

Expand Down Expand Up @@ -46,4 +55,5 @@
{% block activelink %}
$("#navbar-project-menu").addClass("active");
$("#navbar-resource").addClass("active");

{% endblock %}
Loading