-
Notifications
You must be signed in to change notification settings - Fork 1
/
learning_google_cloud.md.rkt
1759 lines (1157 loc) · 89.4 KB
/
learning_google_cloud.md.rkt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#lang scribble/text
@(require "scribble-utils.rkt")
---
path: /learnings/learning_google_cloud
title: Learning Google Cloud
---
# Table of contents
<!-- toc -->
- [Basics](#basics)
- [Cloud Console](#cloud-console)
- [Cloud IAM](#cloud-iam)
* [Roles](#roles)
* [Service Accounts](#service-accounts)
- [Cloud SDK](#cloud-sdk)
- [Tools](#tools)
- [Monitoring](#monitoring)
* [Billing](#billing)
- [Networking](#networking)
* [VPC](#vpc)
<!-- tocstop -->
# Basics
Terms:
* Project -- can not create resources that span projects. In some cases you can create shared resource. DELETING PROJECT WILL DELETE ALL RESOURCES TO IT (aka: you've experimented and want to make sure you don't get billed for that huge GKE cluster? Just delete the project!)
- can be assigned to folders ("location" when creating the project)
* Zone -- 2+ data centers co-located, likely closely enough. 1ms round trip. Independent failure domain
* Region -- geographic area, 2 or more zones seperated by tens of miles. 5ms latency
* Project ID -- globaly unique (Google will _make_ / preview this unique when you create a project name)
# Cloud Console
https://console.cloud.google.com
# Cloud IAM
domain / organization -> folders -> roles -> principles (the humans)
includes default service accounts as starting point / best practice!
## Policies
policies are inherited from parents
a less restricted parent policy will override a more restricted resource policy
policies are set on a resource
each policy contains a set of roles and members
## Roles
Viewer
Editor <-- edit stuff
Owner <-- modify privs
(these are not great from a least priviledged aspect, but they're an AppEngine thing that came forward)
this is on the test!
[BIG OLD LIST OF ROLES ACROSS ALL PRODUCTS](https://cloud.google.com/iam/docs/understanding-roles#cloud-domains-roles)
## Service Accounts
right there in the IAM sidebar
# Cloud SDK
# Tools
## Cloud Shell / Cloud Shell Editor
Cloud Shell Editor <-- VS Code running on the web
Cloud Shell <-- CLI and this is authed based on the project you're logged into. BUT is persistent!
## CLI tools for managing GCP resources etc
Avail through Cloud SDK.
gcloud
gsutil
### awesome gcloud tricks
#### configurations to manage multiple projects, etc
Can use [configurations](https://cloud.google.com/sdk/docs/configurations) to jump frome one set of settings to another. AKA set the default project, etc.
$ gcloud config configurations create my-new-config
$ gcloud config set project my-latest-project # will be configuration specific now!
$ gcloud config configurations list # show all of what I have, including active or not information
$ gcloud config configurations activate default # go back
Can also have this set through environmental variable by setting `CLOUDSDK_ACTIVE_CONFIG_NAME` ie through `direnv` or something
Super sloppy way to get just the name of the active config: ` gcloud config configurations list | grep True | cut -f 1 -d ' '`
Then you may want to do things like `gcloud container clusters get-credentials` to set third party tools like `kubectl` correctly.
##### See also
* GCP_GKE_Kubectl
## How much of my resource quotas am I using?
[IAM Quota tool](https://console.cloud.google.com/iam-admin/quotas)
## Which APIs / services are currently enabled?
https://console.cloud.google.com/apis/dashboard
# Monitoring
"Cloud Monitoring": formerly known as "StackDriver" (used to be a third party company, not acquired)
## Billing
Can do billing data -> bigquery so can SQL QUERY FOR IT!!!!
See GCP_BigQuery
## Log Explorer
[Log Explorer Query Syntax Language Guide](https://cloud.google.com/logging/docs/view/logging-query-language)
# Networking
## VPC
Global resource!!!!! (not region based like AWS)
Elements:
* default <-- created for you. Probably don't want this b/c you don't control this
* automode <-- can not connect to other VPCs with VPC peering, allows traffic from subnets and to/from internet. (Uggh)
* custom mode <-- define CIDR blocks, define firewall rules to open up just what you need
* Subnets <-- regional resources
* Routes
* Firewall rules (ingest, egress)
* VPC flow logging
* VPC peering
* shared VPC <-- lets you share VPCs with other project
automode VPC - includes one subnet per region
### See also
* [VCP how-to guide](https://cloud.google.com/vpc/docs/how-to)
## Subnets
## Load Balancing
Hooks up Backend configuration - allows you to create a Service entry and configure health checks for that service etc etc.
## Static IPs
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{To create a new IP reserve, click on Reserve Static Address and fill in the details. In the wizard, you have the option to select the service tier type, region, and, most important, the VM instance that will use it.}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{any public IP address that a VM instance is using is not static, and when the instance is restarted, it gets a new IP address.}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Under Type, you will see if the IP is ephemeral or static. An ephemeral IP will get replaced on reboot, while static will stay}
## Cloud DNS
Global scope only
Public or private zones
Private zones 1:1 with VPC network
DNS peering for cross-network resolution
Uses Cloud Domains for DNS registration
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{The zone name is a unique ID inside Google Cloud that is similar to a Compute Engine instance ID or a Cloud Bigtable instance ID. The DNS name is specific to the domain name system and refers to the subgroup of records for which this zone acts as a delegate.}
## Firewalls
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Firewall policies allow you to expand the GCP firewall service and apply rules at the organization and folder levels.}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{The GCP firewall offering is separated into the following services:
VPC firewall: This firewall helps you create firewall rules that apply to your VPC.
Firewall policies: These policies can be applied on a folder, project, or organization level.
Firewall rules for App Engine: These rules control traffic into our App Engine applications.
Firewall Insights: Gives detailed information on the logs of your firewall rules and traffic that is passing into and out of the firewall.}
# Compute Services
## Compute Engine (GCE)
Zonal resource
Live migration - virtual machines moved to different hardware while running
When you make one of these by clicking around you can copy the `gcloud` construction CLI parameters from the console!!!
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Using the Container feature, you can specify a GCR container image that you would like to run on the VM without needing to run a single command like docker run. This feature is great if you need to run a single container image on your VM directly from GCR.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{setting a shutdown script—sort of the opposite of what you did with your instance templates’ startup scripts. Once the termination is triggered, GCE gives the VM 30 seconds to finish up, and then sends a firm termination signal (the equivalent of pressing your machine’s power button) and switches the machine to terminated}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{images are a good starting point for your VMs, and although you can create custom images, the curated list that Google provides should cover the common scenarios}
### Instance Types
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{By default, the [Google Container Optimized] image is configured to download updates every week, keeping it secure and optimized all the time.
}
### administration
Identify aware proxy: allows you to SSH into an instance without needing to open up SSH. So a cloud jumpbox. The best practice for connecting to your instance.
(or you could open the port, but that's a slight security risk etc etc)
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Using a snapshot schedule, you can configure the backup process to take place on specific days and times.}
### monitoring
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{ In GCP, you monitor VM instances using a Cloud Monitoring agent, which is a software daemon that collects metrics regarding the performance of the VM and passes them to Cloud Monitoring.}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{The agent provides a deeper insight into the VM.}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{curl -sSO https://dl.google.com/cloudagents/add-monitoring-agent-repo.sh
$ sudo bash add-monitoring-agent-repo.sh
$ sudo apt-get update
$ sudo apt-get install stackdriver-agent}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Cloud Monitoring agent, you need to create a service account with enough permissions to read and write events from the VM instance to Cloud Monitoring}
### Instance groups
multiple GCE instances grouped together. Integration with auto scaling, LBs.
Managed <-- you upload an instance template and GCP manages the herd. Also has stateful option.
unmanaged <-- you add pre-created VMs to the group
## Cloud Run
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{With Vertical Pod Autoscaling (VPA), GKE handles the resource allocation of pods and automatically scales pods up and down.}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Cloud Run will scale applications horizontally, which means that GCP will add more pods to the application and will not add more CPU or RAM to existing pods.}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Domain mapping can map a top-level domain to a Cloud Run application. To do just that, from the Cloud Run console, click on the “Manage Custom Domains” link}
## GKE
<<Google_Cloud_Platform_Kubernetes_Engine>>
likely this stuff will be on the test too!
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Anthos allows you to turn your GKE cluster to Cloud Run–enabled infrastructure, which means that you can deploy your workload to GKE with the Cloud Run tools. }
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{To allow external access to your app, you need to explicitly expose it. You do so on GKE using the kubectl expose command, as you can see in the following code:}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{GCP has developed its own Linux image for Kubernetes nodes, called Container-Optimized OS (cos), and it is set as the default option. }
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{GCP has developed its own Linux image for Kubernetes nodes, called Container-Optimized OS (cos), and it is set as the default option. }
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{NodeA GKE cluster’s master node and nodes can run different versions of Kubernetes.}
### scaling considerations
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{It is highly important and recommended you use VPA with GKE Cluster Autoscaler otherwise your cluster will run out of resources.}
@quote-highlight[#:title "Getting Started with Containers in Google Cloud Platform : Deploy, Manage, and Secure Containerized Applications"
#:author "Ifrah, Shimon"
#:page-number 0]{Autoscaler’s decision to scale resources is also based on the auto-scaling profile you configure. Scaling profiles are based on the following two profiles:
Balanced: This is the default profile that GKE will use when using Autoscaler.
Optimize-utilization: This profile is very cost oriented and will try to keep the cost of running your cluster as low as possible by scaling down the cluster as soon as possible; however, this is not recommended for most environments, because of the aggressive nature of the profile, which prioritizes cost and not performance.}
### Neat GKE operator hacks
If you go into a pod in the toolbar / menu bar there is a `KUBECTL` dropdown. This will let you - in addition to other things - attach to the running pod in the Google Cloud Shell web thing.
### and kubectl <<GCP_GKE_Kubectl>>
### How many IP addresses do I need / what should my CIDR block look like?
Your IP address range [corresponds to how many nodes you need](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_pods), but not as directly as you might think. see [IP address ranges for VPC native clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing) but also note that K8s reserves IP addresses for recently turned off pods (it will not instantly recycle those pods, assuming there may be traffic still flowing)
### and specialized networking concerns
> For example, in Google Cloud, any traffic to the internet must come from a VM's IP. When containers are used, as in Google Kubernetes Engine, the Pod IP will be **rejected** for egress. To avoid this, we must hide the Pod IP behind the VM's own IP address - generally known as "masquerade"
Most IPs are masquerade-ed **EXCEPT** these CIDR blocks:
* 10.0.0.0/8
* 172.168.0.0/12
* 192.168.0.0/16
[Source: IP-MASQ-AGENT](https://kubernetes.io/docs/tasks/administer-cluster/ip-masq-agent/)
If you need to talk to IPs within these ranges - like for example you're talking to another Google hosted cloud solution via some kind of peer VPC thing - you may need to force a range ON.
Ways to do this:
* [ip-masq-agent](https://github.com/kubernetes-sigs/ip-masq-agent) <-- 500 lines of Go code on top of IPTables
* [k8s-custom-iptables](https://github.com/bowei/k8s-custom-iptables) <-- 100 lines of Bash on top of IPTables
This works at all because [kube-proxy currently uses iptables under the hood](https://itnext.io/kubernetes-service-load-balancing-kube-proxy-and-iptables-da3ebf1c802a). Which works on both incoming and outbound traffic.
#### See Also
* [Configuring an IP masquerade agent](https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent)
* [K8s Networking demystified: a brief guide](https://www.stackrox.io/blog/kubernetes-networking-demystified/)
### Monitoring / Operating
Resource consumption monitoring: (in the TF plugin this defaults to `true`)
### Config Connector
It puts a K8s CRD interface over constructing resources in GCP.
#### How to ask K8s for the documentation on Config Connector CRDs
`kubectl describe crd iampartialpolicies.iam.cnrm.cloud.google.com`
#### Referencing cross project resources
(at this writing - May 2023 - this seems to be only documented in the OpenAPI spec for the CRDs...)
The error:
Upgrade "THING" failed: failed to create resource: IAMPartialPolicy.iam.cnrm.cloud.google.com "THING" is invalid: [<nil>: Invalid value: "": "spec.resourceRef" must validate one and only one schema (oneOf). Found none valid, <nil>: Invalid value: "": "spec.resourceRef" must not validate the schema (not)]
So, to reference a cross-project resource, here's the `resource` chunk of YAML to use - targetting some Spanner database that happens to live in another project
- resource:
apiVersion: spanner.cnrm.cloud.google.com/v1beta1
kind: SpannerDatabase
external: "projects/MY-OTHER-PROJECT/instances/SPANNER-INSTANCE-NAME/databases/MY-DB"
role: "roles/spanner.databaseReader"
Q: what are the allowed values for "external"?
A: Well, for IAMPolicy or IAMPartialPolicies the [format is documented here](https://cloud.google.com/config-connector/docs/reference/resource-docs/iam/iampolicy#supported_resources)
#### IAMPartialPolicy
[IAMPartialPolicy](https://cloud.google.com/config-connector/docs/reference/resource-docs/iam/iampartialpolicy) seems to only exist for Cloud Connector (ie it does not exist for Terraform).
> represents a non-authoritative intent for the associated Google Cloud resource's IAM policy bindings. Config Connector merges the bindings in the IAMPartialPolicy spec with the bindings and audit configs that already exist in the underlying IAM policy
But it's helpful if you have to create more than one definition of policies... (but also _are you sure_?)
#### Common Errors
`resource reference for kind must include API group`
If you `resourceRef` an object, make sure your `resourceRef` has an `apiVersion` field and the value of that field matches the apiVersion of the resource in question.
it is not an error on the resource in question, but the reference to that resource...
## Cloud Functions
upload from web based editor, zip file upload, cloud source repository
Has a testing tab so you can try to invoke the function directly.
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Google Cloud Functions is a serverless compute solution that allows you to run event-based applications}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Cloud Functions comes with a perpetual free tier, which means that some chunks of the resources you use are completely free. With Cloud Functions, the following numbers represent free-tier usage and won’t count towards your bill:
Requests—the first 2 million requests per month
Compute—200,000 GHz-seconds per month
Memory—400,000 GB-seconds per month
Network—5 GB of egress traffic per month}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{gcloud beta functions deploy echo \
> --source=https://source.developers.google.com/
projects/your-project-id-here/repos/echo \ 1
> --trigger-http}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{gcloud tool has a call function that triggers a function and allows you to pass in the relevant information. This method executes the function and passes in the data that would have been sent by the trigger, so you can think of it a bit like an argument override.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{gcloud beta functions deploy echo --source=./echo/ \
--trigger-http --stage-bucket=my-cloud-functions}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Even though events from different sources share quite a bit in common, they fall into two categories. Events based on HTTP requests are synchronous events (the requester is waiting for a response), whereas those coming from other services such as Cloud Pub/Sub are asynchronous (they run in the background).}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{The most common event that you’re likely familiar with is an HTTP request, but they can also come from other places such as Google Cloud Storage or Google Cloud Pub/Sub. Every event comes with attributes, which you can use when building your function, including the basic details of an event (such as a unique ID, the type of the event, and a timestamp of when the event occurred), the resource targeted by the event, and the payload of data specific to the event}
### Node.js specific notes
To run locally:
Point `--source` to your ie built code and `--target` to the Express middleware compatible property / function object you want to run.
example command:
`npx @at-sign{}google-cloud/functions-framework --source=dist/main/index.js --target=exportedPropertyInIndexThatCouldBeAFunction`
ES modules are supported. Two conditions:
1. The property you reference must be exported from whatever you've told `function-framework` to `--source`
2. It can be an exported [Express middleware compatible function](https://medium.com/google-cloud/express-routing-with-google-cloud-functions-36fb55885c68) (_not_ a function that returns a Express Router)
#### "But I want to serve multiple endpoints from my one GCP Cloud Function?"
For the normal case, of one endpoint per function, the docs say to do this (pardon the Typescript):
```typescript
import * as ff from '@at-sign{}google-cloud/functions-framework';
export function doIt(req: ff.Request, res: ff.Response) {
res.send('hi')
}
```
Then use `npx @at-sign{}google-cloud/functions-framework --target doIt` to execute the function
In a use case of multiple routes handled by the cloud function, make use of the fact that Cloud Functions are Express compatible middleware, from point 2 above
We'll get fancy here and wrap this up into a function
```typescript
export class CloudApp {
static get doIt() {
const app = express()
app.get("/hi", (req, res) => res.send('hi'))
app.get("/hello", (req, res) => res.send("hello"))
return app
}
}
```
Then use `npx @at-sign{}google-cloud/functions-framework --target CloudApp.doIt` to execute the function
Google's cloud function target parameter really expects a property, not calling a function. But we want to be able to add routes at runtime!
So make a class, with a static getter function, and all that fancy means we have dynamic property - when ES6+ requests the "app" property it automatically executes the function found there and returns the result as the value of the property.
### Deploying Cloud Functions
[Relatively simplistic answer for this](https://ryderdamen.com/blog/how-to-deploy-gcf-from-circleci/)
#### when you use private Artifact Registries
If they are in separate projects you need to give the default cloud build service account Artifact Reader to that repo. Note that this is NOT the service account you're calling gcloud functions deploy with, as gcloud does some fancy use another service account stuff.
cloud functions deploy spits out serviceConfig YAML which lists serviceAccountEmail. This seems to be the default cloud build service account you need
Alternatively, just specify the authorization key for Cloud Build to use, depending on the fact that the Cloud Functions are (now?) just a layer over Cloud Build + Cloud Run
Source:
* [GCP Artifact Registry documentation on Cloud Functions](https://cloud.google.com/artifact-registry/docs/integrate-functions)
* [Injecting NPM authentication key into Cloud Build](https://cloud.google.com/blog/topics/developers-practitioners/using-private-repo-artifact-registry-google-cloud-functions)
## App Engine
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{If you wanted to split 50% of the traffic currently going to version-a, you could do this by clicking the Split Traffic icon (which looks like a road sign forking into two arrows), which brings you to a form where you can configure how to split the traffic}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{App Engine uses some of the memory (about 0.4 GB) for overhead on your instance.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{By default (if you leave these fields out entirely), you’ll get a single-core VM with 0.6 GB of RAM, }
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{App Engine Standard involves running your code in a special sandbox environment, you’ll need a way of configuring the computing power of that environment. To do so, you’ll use a setting called instance_class in your app.yaml file. }
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{By default, Flex services are limited to 20 instances, but you can increase or decrease that limit.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{As you’ve learned, at least one instance must be running at all times, but it’s recommended to have a minimum of two instances to keep latency low in the face of traffic spikes}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Similar to App Engine Standard, Flex has two scaling options: automatic and manual. The only difference is that Flex is lacking the “basic” scaling option. }
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Basic scaling has only two options to control, which are the maximum number of instances and how long to keep an idle instance around before turning it off.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{By default, App Engine will aim to handle eight requests concurrently on your instances}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{App Engine allows you to set a concurrency level as a way to trigger turning more instances on or off, meaning you can set a target for how many requests an instance can handle at the same time before it’s considered too busy. }
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{The minimum pending latency is the way you set a lower bound when telling App Engine when it’s OK to turn on more instances. When you set a minimum pending latency, it tells App Engine to not turn on a new instance if requests aren’t waiting at least a certain amount of time. }
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Now that you have a simple Dockerfile that serves some content, the next thing you’ll need to do is update your app.yaml file to rely on this custom runtime. To do that, you’ll replace nodejs in your previous definition with custom. This tells App Engine to look for a Dockerfile}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{The main difference you’ll notice at first is that it takes a bit longer to complete the deployment. It takes more time primarily because App Engine Flex builds a Docker container from your application code, uploads it to Google Cloud, provisions a Compute Engine VM instance, and starts the container on that instance.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Once you see that the new version works the way you expect, you can safely promote it by migrating all traffic to it using the Cloud Console.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{You can update the code again to change the message and deploy another version, without it becoming the live version immediately. To do so, you’ll set the promote_by_default flag to false:}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{It turns out that just as you can access a specific service directly, you can access the previous version by addressing it directly in the format of <version>.<service>.your-project-id-here.appspot.com (or using -dot- separators for HTTPS):}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{if you host lots of versions concurrently, those versions will spawn instances as necessary to service the traffic. If they’re running inside App Engine Flex, each version will have at least one VM running at all times.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Flex applications must always have at least a single VM instance running. As a result, Flex applications end up costing money around the clock. }
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Services on App Engine provide a way to split your application into smaller, more manageable pieces. }
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Each of your projects is limited to one application, with the idea that each project should have one purpose}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{App Engine uses four organizational concepts to understand more about your application: applications, services, versions, and instances}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{App Engine Flexible Environment (often called App Engine Flex) provides a fully managed environment with fewer restrictions and somewhat more portability, trading some scalability in exchange. App Engine Flex is based on Docker containers}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{App Engine Standard Environment, released in early 2008, offers a fully managed computing environment complete with storage, caching, computing, scheduling, and more.}
### Classic
Scans to 0
BUT with some severe limits.
Languages: Python, Java, PHP, Ruby, Go, Node
### Flexible
min footprint - no scaling to zero here.
## Load Balancing
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{To create the load balancer, choose Network Services from the left-side navigation in the Cloud Console, and choose Load Balancing after that.}
## Cloud CDN
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Cloud CDN sits between the load balancer and the various people making requests to the service and attempts to short-circuit requests}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{By default, Cloud CDN will attempt to cache all pages that are allowed. This definition mostly follows IETF standards (such as RFC-7234), meaning that the rules are what you’d expect if you’re familiar with HTTP caching in general. For example, the following all must be true for Cloud CDN to consider a response to a request to be cacheable:
Cloud CDN must be enabled.
The request uses the GET HTTP method.
The response code was “successful” (for example, 200, 203, 300).
The response has a defined content length or transfer encoding (specified in the standard HTTP headers).
In addition to these rules, the response also must explicitly state its caching preferences using the Cache-Control header (for example, set it to public) and must explicitly state an expiration using either a Cache-Control: max-age header or an Expires header.
Furthermore, Cloud CDN will actively not cache certain responses if they match other criteria, such as
The response has a Set-Cookie header.
The response size is greater than 10 MB.
The request or response has a Cache-Control header indicating it shouldn’t be cached (for example, set to no-store).}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{This is a common scenario when, for example, you deploy new static files, such as an updated style.css file, and don’t want to wait for the content to expire from the cache.
To do this, you can use the Cloud Console and click the Cache Invalidation tab. }
# Storage
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Cloud Storage. Google Cloud provides the following storage options:
Zonal standard persistent disk and zonal SSD persistent disk
Regional standard persistent disk and regional SSD persistent disk
Local SSD for high-performance local block storage
Cloud Storage buckets: Object storage.
Filestore: High-performance file storage
}
## Cloud Storage
Needs to be **globally** unique. Ideally workloads that are write once read many (as you have to replace objects to update them)
Lots of different options here that balance price, access time and min retention policy:
* Standard
* Nealline
* Farline
* Archive
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{each file in the bucket must not be larger than 5 terabytes}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{change notifications allow you to set a URL that will receive a notification whenever objects are created, updated, or deleted. Then you can do whatever other processing you might need based on the notification.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{You can define a couple of conditions to determine when objects should be automatically deleted in your bucket:
* IsLive
* NumberOfNewVersions
* CreatedBefore
* Age
}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Owners can change ACLs and metadata, which means that unless you trust someone to grant further access appropriately, you shouldn’t give them the Owner permission.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Signed URLs take an intent to do an operation (for example, download a file) and sign that intent with a credential that has access to do the operation}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{the more common scenario where you have content in your app that you want to share temporarily with users? For example, you might want to serve photos, but you don’t want them always available to the public to discourage things like hotlinking. Luckily this is easy to do in code}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{In any version-enabled bucket, every object will have a generation (tracking the object version) along with a metageneration (tracking the metadata version)}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{deleting objects from versioned buckets because deleting the file itself doesn’t delete other generations.}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{ If you want to remove the file along with all of its previous versions, pass the -a flag to the gsutil rm command:}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{object-based storage uses a flat namespace to store your data. The key functionality of the Cloud Console interface is that it translates this flat namespace to replicate a folder hierarchy. Cloud Storage has no notion of folders.
}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{object-based storage uses a flat namespace to store your data. The key functionality of the Cloud Console interface is that it translates this flat namespace to replicate a folder hierarchy. Cloud Storage has no notion of folders.
}
### and bucket set retention period
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{If you attempt to delete objects younger than the retention period it will result in a PERMISSION_DENIED error.}
### and PII auditing
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{We can loop through all the buckets and objects, then scan them for PII information with the Google Cloud Data Loss Prevention API.}
@quote-highlight[
#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{A bucket lock allows you to create a retention policy that locks the data preventing it from being deleted or overwritten. You can also lock a retention policy. Once it is locked you cannot unlock it; you will only be able to increase the retention period}
### signed URLs
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Signed URLs is a URL that provides access to users and applications for a limited time. The signed URL allows users to access the object without authentication.}
# Cloud Dataflow
Managed Apache Beam
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Apache Beam is a programming model that defines and executes the defined pipeline. The pipelines can be batch and streaming which are exposed to different runners as:
Google Cloud Dataflow
Apache Spark
Apache Flink
Apache Apex
DirectRunner (a local runner for testing)}
## Terms
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{A pipeline defines what steps the runner will execute on your data}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{A PCollection defines the data on which your pipeline will operate on}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{A transform is a function that you define that is performed on your data}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{A ParDo is an Apache Beam transform operation. As outlined in the Transforms section, it performs a user defined operation on a collection of elements. The output of a ParDo can be a single element or many elements, however, it does not output a single output per input element. }
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{A Map is another transform operation available in Apache Beam. In the Framework you will be using the beam.Map as you will be performing a one-to-one mapping,}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Apache Beam I/O connectors let you read/write data into your pipeline as well as write output data}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{As an example, the Framework you are working with has a source of Cloud Pub/Sub and a sink of BigQuery}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Aggregation is an operation that is performed on many elements to produce some grouped value from those respective elements.}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Apache Beam provides the following aggregation methods:
CoGroupByKey
CombineGlobally
CombinePerKey
CombineValues
Count
Distinct
GroupByKey
GroupBy
GroupIntoBatches
Latest
Max
Min
Mean
Sample
Sum
Top}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Runners are the software that accepts a pipeline and executes it.}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Windowing enables you to group operations over the unbounded data set by dividing the data set into windows of finite collections according to their timestamps of the individual elements. You set the following windows with the Apache Beam SDK or Dataflow SQL streaming extensions:
Tumbling windows (called fixed windows in Apache Beam)
Hopping windows (called sliding windows in Apache Beam)
Session windows
}
# Pub Sub
From [Cloud Pub/Sub Documentation](https://cloud.google.com/pubsub/docs/overview)
> Pub/Sub enables you to create systems of event producers and consumers, called publishers and subscribers. Publishers communicate with subscribers asynchronously by broadcasting events, rather than by synchronous remote procedure calls (RPCs).
> Pub/Sub adopts from messaging middleware is per-message parallelism (rather than partition-based). Pub/Sub "leases" individual messages to subscriber clients, then keeps track of whether a given message has been successfully processed.
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Each subscription sees all of the messages you send on a topic}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Once one consumer consumes a message from a subscription, that message is no longer available on that same topic, so the next consumer will get a different message}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Work-queue messaging
Unlike fan-out messaging, where you deliver each message to lots of consumers, work-queue messaging is a way of distributing work across multiple consumers, where, ideally, only one consumer processes each message}
@quote-highlight[#:title "Programming Google Cloud"
#:author "Rui Costa"
#:page-number 0]{Pub/Sub does not guarantee the order of message delivery.}
Can view the messages in the web console UI for debugging purposes. Can also ACK here tooo!
Also PubSub Lite.
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{the payload is always base-64 encoded, whereas attributes aren’t,}
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{Once Cloud Pub/Sub receives the message, it’ll assign the message an ID that’s unique to the topic and will return that ID to the producer as confirmation that it received the message (figure 21.1)}
## Core concepts
* Publisher
* Topic
* Subscription <-- A topic can have multiple subscriptions, but a given subscription belongs to a single topic.
* Subscriber <-- program or instance of listening to / consuming a subscription
* Message
* Message Attribute <-- Messages are k/v pairs a publisher can define
* ACK <-- signal sent by a subscriber after it has successfully retrieved (? and processed?) the message
Messages pushed out to all subscriptions in Cloud Pub/Sub AT LEAST once. (aka: yes a subscriber does not pull from a consumer group / partition, essentially each subscription is its own stream)
> Pub/Sub delivers each published message at least once for every subscription.
[Source](https://cloud.google.com/pubsub/docs/subscriber) "at least once delivery section"
Communication can be:
* one to many <-- fan out
* many to one
* many to many
### Messages, sizes and quotas
10 MB/s per open stream. This is not just a quota but the buffer between the service and client library.
There also [may be a limit of 1,000 messages per pull](https://stackoverflow.com/a/58712547), regardless(??)
> To understand the impact of this buffer on client library behavior, consider this example:
>
> There is a backlog of 10,000 1KB messages on a subscription.
> Each message takes 1 second to process sequentially, by a single-threaded client instance.
> The first client instance to establish a StreamingPull connection to the service for that subscription will fill its buffer with all 10,000 messages.
> It takes 10,000 seconds (almost 3 hours) to process the buffer.
> In that time, some of the buffered messages exceed their acknowledgement deadline and are re-sent to the same client, resulting in duplicates.
> When multiple client instances are running, the messages stuck in the one client's buffer will not be available to any client instances.
>
> This would not occur if you are using Flow Control…
[Source](https://cloud.google.com/pubsub/docs/pull#streamingpull_dealing_with_large_backlogs_of_small_messages)
If you publish 10 500-byte messages in separate requests, your publisher quota usage will be 10,000 bytes. This is because messages that are smaller than 1000 bytes are automatically rounded up to the next 1000-byte increment.
[Source](https://cloud.google.com/pubsub/quotas)
So: Small messages (< 1K): 10,000 / second per stream
> If/when the user runs out of throughput quota, the stream is suspended, but the connection is not broken. When there is sufficient throughput quota available again, the connection is resumed.
[Source](https://cloud.google.com/pubsub/docs/pull)
### Streaming data patterns and how to architect them
#### Load balancing
"Load balancing" processing: Create a subscription to a topic and have multiple subscribers subscribe to the same subscription
> Multiple subscribers can make pull calls to the same "shared" subscription. Each subscriber will receive a subset of the messages.
[Source](https://cloud.google.com/pubsub/docs/subscriber#push-subscription) see comparison table
##### And H/A considerations
> To effectively load-balance across all your subscribers when the message load is small, or to achieve the goal of never starving a subscriber when you have a small message load, I would recommend using synchronous pull. Here's an example,
[Source](https://github.com/googleapis/java-pubsub/issues/582#issuecomment-863603463)
#### Fan out
"Fan out" need to do N different things with this message in parallel: create N subscriptions, each with howevever many subscribers. Thus the message goes to every subscription. Say one subscription just cares about user notification, one subscription just cares about bookkeeping, etc.
[Source](https://cloud.google.com/pubsub/architecture#the_basics_of_a_publishsubscribe_service)
### On Delivery methods
Methods:
* Push <-- each message goes to a subscriber defined endpoint
* Pull (your application asks for next message):
* Synchronous pull <-- like pull but more like polling
* Streaming pull <— "pull" as in creating a socket and getting messages as they are available published down the socket
Push: needs to have a public HTTPS endpoint. More latent then pull
Pull: Pull and Streaming Pull (default).
Synchronous pull use cases:
* precise cap on messages sent (streaming pull may oversubscribed for first little bit)
* if you have spikey loads of very small messages
* languages without GRPC
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{With a push subscription, you’ll rely on HTTP response codes to communicate that. In this case, an HTTP code of 204 (No Content) is your way of saying you’ve successfully received and processed the message. Any other code (for example, a 500 Server Error or 404 Not Found) is your way of telling Cloud Pub/Sub that some sort of failure occurred.}
#### Message Control flow (needed aka when you have multiple replicas of a microservice who run the same consumer...)
> It's possible that one client could have a backlog of messages because it doesn't have the capacity to process the volume of incoming messages, but another client on the network does have that capacity.
[Source: Google PubSub documentation](https://cloud.google.com/pubsub/docs/pull#flow_control)
In that case set up a [flow control builder](https://cloud.google.com/pubsub/docs/samples/pubsub-subscriber-flow-settings) and attach it to your subscriber.
### On ACK
@quote-highlight[#:title "Google Cloud Platform in Action"
#:author "Geewax, JJ"
#:page-number 0]{On each subscription, in addition to the push or pull configuration, you also must specify what’s called an acknowledgment deadline. This deadline, measured in seconds, acts as a timer of how long to wait before assuming that something has gone wrong with the consumer}