Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Chunking Support #312

Merged
merged 8 commits into from
Sep 20, 2019
Merged

API Chunking Support #312

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
97ddecf
Initial API chunking support
agrare Dec 3, 2018
897542b
Use refresher_options for chunk size
agrare Dec 6, 2018
fe339d4
Bump kubeclient to 4.1.1
agrare Sep 18, 2019
735b333
Don't include RefreshMixin in empty class
agrare Sep 18, 2019
0f527d9
Update specs for chunking
agrare Sep 19, 2019
5baac4a
Don't match on query to simulate limit not supported
agrare Sep 19, 2019
cd1e6e5
Better gemspec version for kubeclient 4.1
agrare Sep 19, 2019
2b2a7e4
Comment the legacy refresher spec vcr options
agrare Sep 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion app/models/manageiq/providers/kubernetes/container_manager/refresher_mixin.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,18 @@ def parse_targeted_inventory(ems, target, inventory)

def fetch_entities(client, entities)
entities.each_with_object({}) do |entity, h|
continue = nil
h[entity[:name].singularize] ||= []

begin
h[entity[:name].singularize] = client.send("get_#{entity[:name]}")
loop do
entities = client.send("get_#{entity[:name]}", :limit => refresher_options.chunk_size, :continue => continue)

h[entity[:name].singularize].concat(entities)
break if entities.last?

continue = entities.continue
end
rescue KubeException => e
raise e if entity[:default].nil?
$log.warn("Unexpected Exception during refresh: #{e}")
Expand Down
1 change: 1 addition & 0 deletions config/settings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@
:kubernetes:
:refresh_interval: 15.minutes
:streaming_refresh: false
:chunk_size: 1_000
cben marked this conversation as resolved.
Show resolved Hide resolved
:inventory_object_refresh: true
:inventory_collections:
:saver_strategy: batch
Expand Down
2 changes: 1 addition & 1 deletion manageiq-providers-kubernetes.gemspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Gem::Specification.new do |s|

s.add_runtime_dependency("hawkular-client", "~> 4.1")
s.add_runtime_dependency("image-inspector-client", "~> 2.0")
s.add_runtime_dependency("kubeclient", "~> 4.0")
s.add_runtime_dependency("kubeclient", "~> 4.1.1")
cben marked this conversation as resolved.
Show resolved Hide resolved
s.add_runtime_dependency("prometheus-alert-buffer-client", "~> 0.2.0")
s.add_runtime_dependency("prometheus-api-client", "~> 0.6")
s.add_runtime_dependency("more_core_extensions", "~> 3.6")
Expand Down
3 changes: 2 additions & 1 deletion spec/models/manageiq/providers/kubernetes/container_manager/refresh_mixin_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

describe ManageIQ::Providers::Kubernetes::ContainerManager::RefresherMixin do
let(:client) { double("client") }
let(:dummy) { (Class.new { include ManageIQ::Providers::Kubernetes::ContainerManager::RefresherMixin }).new }
cben marked this conversation as resolved.
Show resolved Hide resolved
let(:ems) { FactoryBot.create(:ems_kubernetes) }
let(:dummy) { ManageIQ::Providers::Kubernetes::ContainerManager::Refresher.new([ems]) }

context 'when an exception is thrown' do
before { allow(client).to receive(:get_pods) { raise KubeException.new(0, 'oh-no', nil) } }
Expand Down