Add a GUC to log HTTP traffic #71

sfc-gh-okalaci · 2025-11-19T12:03:55Z

Especially useful for #68

An example from #68:

INFO:  making POST request to URL http://localhost:5433/api/catalog/v1/oauth/tokens : {grant_type=client_credentials&scope=PRINCIPAL_ROLE:ALL}
INFO:  received response with status code 200, body: {"access_token":"****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************","token_type":"bearer","issued_token_type":"urn:ietf:params:oauth:token-type:access_token","expires_in":3600}
INFO:  making GET request to URL http://localhost:5433/api/catalog/v1/postgres/namespaces/public/tables/t_rest_2
INFO:  received response with status code 200, body: {"metadata-location":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00012-dfe4f634-378c-44ae-a724-ad3ffa84bf72.metadata.json","metadata":{"format-version":2,"table-uuid":"3ef2737c-a0b8-4f92-bf69-1d92be9264c2","location":"s3://okalaci/postgres/public/t_rest_2/1480508","last-sequence-number":11,"last-updated-ms":1763631718360,"last-column-id":3,"current-schema-id":2,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"int"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"a","required":false,"type":"int"},{"id":2,"name":"b","required":false,"type":"int"}]},{"type":"struct","schema-id":2,"fields":[{"id":1,"name":"a","required":false,"type":"int"},{"id":2,"name":"b","required":false,"type":"int"},{"id":3,"name":"c","required":false,"type":"int"}]}],"default-spec-id":1,"partition-specs":[{"spec-id":0,"fields":[]},{"spec-id":1,"fields":[{"name":"c_bucket_3","transform":"bucket[3]","source-id":3,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{},"current-snapshot-id":6153398898562664325,"refs":{"main":{"snapshot-id":6153398898562664325,"type":"branch"}},"snapshots":[{"sequence-number":6,"snapshot-id":7756046587847404627,"parent-snapshot-id":4067525868888936465,"timestamp-ms":1763554473987,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-7756046587847404627-1-6b6ad6d4-2297-4f2c-a200-c42740e1e357.avro","schema-id":1},{"sequence-number":7,"snapshot-id":7836393630613138013,"parent-snapshot-id":7756046587847404627,"timestamp-ms":1763631128811,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-7836393630613138013-1-f8b20bbf-6d98-427e-a985-e3112b1d2ba1.avro","schema-id":2},{"sequence-number":8,"snapshot-id":3825740701725564565,"parent-snapshot-id":7836393630613138013,"timestamp-ms":1763631187018,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-3825740701725564565-1-8be5321a-944c-41a0-83bc-a51d9bd9ed3c.avro","schema-id":2},{"sequence-number":9,"snapshot-id":8678269831436150991,"parent-snapshot-id":3825740701725564565,"timestamp-ms":1763631228367,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-8678269831436150991-1-7b839038-4629-4f9d-84d6-617fd79179fe.avro","schema-id":2},{"sequence-number":10,"snapshot-id":5296707806511941211,"parent-snapshot-id":8678269831436150991,"timestamp-ms":1763631248250,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-5296707806511941211-1-ab4ef520-5c22-4688-be30-8f5c7baa384f.avro","schema-id":2},{"sequence-number":11,"snapshot-id":6153398898562664325,"parent-snapshot-id":5296707806511941211,"timestamp-ms":1763631718360,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-6153398898562664325-1-7fb8f0b9-1a40-48a1-8ed7-10fd8f77e433.avro","schema-id":2}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1763554473987,"snapshot-id":7756046587847404627},{"timestamp-ms":1763631128811,"snapshot-id":7836393630613138013},{"timestamp-ms":1763631187018,"snapshot-id":3825740701725564565},{"timestamp-ms":1763631228367,"snapshot-id":8678269831436150991},{"timestamp-ms":1763631248250,"snapshot-id":5296707806511941211},{"timestamp-ms":1763631718360,"snapshot-id":6153398898562664325}],"metadata-log":[{"timestamp-ms":1763554064906,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00000-9b095d6e-dc8c-46ab-a2eb-8c8a0e6babf7.metadata.json"},{"timestamp-ms":1763554064893,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00001-5e523d63-d1d4-4ab2-917e-043425044b33.metadata.json"},{"timestamp-ms":1763554198984,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00002-fafd6564-6e8b-4e8f-8296-4e3247c1401f.metadata.json"},{"timestamp-ms":1763554259988,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00003-06961f84-c43e-4fa7-8b95-56adb6c28e1e.metadata.json"},{"timestamp-ms":1763554279635,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00004-b7c5d99e-a344-4506-8e33-eb17721897ac.metadata.json"},{"timestamp-ms":1763554283523,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00005-32f36809-8b84-45df-9380-f5a929a56e1b.metadata.json"},{"timestamp-ms":1763554473987,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00006-906a269a-3edc-44d9-93dc-361004732510.metadata.json"},{"timestamp-ms":1763556374007,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00007-8093a079-2df4-4081-8d18-d486a7edd986.metadata.json"},{"timestamp-ms":1763631128811,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00008-7fa3943f-7d49-4390-889d-6336251928e5.metadata.json"},{"timestamp-ms":1763631187018,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00009-f995af6a-f978-4c47-950f-d1ff25de1b39.metadata.json"},{"timestamp-ms":1763631228367,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00010-258198be-4f84-44f0-a7c4-34f0889698ac.metadata.json"},{"timestamp-ms":1763631248250,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00011-b9e513c3-8cb8-4402-9bf9-342528f6c814.metadata.json"}]}}
INFO:  making POST request to URL http://localhost:5433/api/catalog/v1/oauth/tokens : {grant_type=client_credentials&scope=PRINCIPAL_ROLE:ALL}
INFO:  received response with status code 200, body: {"access_token":"****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************","token_type":"bearer","issued_token_type":"urn:ietf:params:oauth:token-type:access_token","expires_in":3600}
INFO:  making GET request to URL http://localhost:5433/api/catalog/v1/postgres/namespaces/public/tables/t_rest_2
INFO:  received response with status code 200, body: {"metadata-location":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00012-dfe4f634-378c-44ae-a724-ad3ffa84bf72.metadata.json","metadata":{"format-version":2,"table-uuid":"3ef2737c-a0b8-4f92-bf69-1d92be9264c2","location":"s3://okalaci/postgres/public/t_rest_2/1480508","last-sequence-number":11,"last-updated-ms":1763631718360,"last-column-id":3,"current-schema-id":2,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"int"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"a","required":false,"type":"int"},{"id":2,"name":"b","required":false,"type":"int"}]},{"type":"struct","schema-id":2,"fields":[{"id":1,"name":"a","required":false,"type":"int"},{"id":2,"name":"b","required":false,"type":"int"},{"id":3,"name":"c","required":false,"type":"int"}]}],"default-spec-id":1,"partition-specs":[{"spec-id":0,"fields":[]},{"spec-id":1,"fields":[{"name":"c_bucket_3","transform":"bucket[3]","source-id":3,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{},"current-snapshot-id":6153398898562664325,"refs":{"main":{"snapshot-id":6153398898562664325,"type":"branch"}},"snapshots":[{"sequence-number":6,"snapshot-id":7756046587847404627,"parent-snapshot-id":4067525868888936465,"timestamp-ms":1763554473987,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-7756046587847404627-1-6b6ad6d4-2297-4f2c-a200-c42740e1e357.avro","schema-id":1},{"sequence-number":7,"snapshot-id":7836393630613138013,"parent-snapshot-id":7756046587847404627,"timestamp-ms":1763631128811,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-7836393630613138013-1-f8b20bbf-6d98-427e-a985-e3112b1d2ba1.avro","schema-id":2},{"sequence-number":8,"snapshot-id":3825740701725564565,"parent-snapshot-id":7836393630613138013,"timestamp-ms":1763631187018,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-3825740701725564565-1-8be5321a-944c-41a0-83bc-a51d9bd9ed3c.avro","schema-id":2},{"sequence-number":9,"snapshot-id":8678269831436150991,"parent-snapshot-id":3825740701725564565,"timestamp-ms":1763631228367,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-8678269831436150991-1-7b839038-4629-4f9d-84d6-617fd79179fe.avro","schema-id":2},{"sequence-number":10,"snapshot-id":5296707806511941211,"parent-snapshot-id":8678269831436150991,"timestamp-ms":1763631248250,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-5296707806511941211-1-ab4ef520-5c22-4688-be30-8f5c7baa384f.avro","schema-id":2},{"sequence-number":11,"snapshot-id":6153398898562664325,"parent-snapshot-id":5296707806511941211,"timestamp-ms":1763631718360,"summary":{"operation":"append"},"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-6153398898562664325-1-7fb8f0b9-1a40-48a1-8ed7-10fd8f77e433.avro","schema-id":2}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1763554473987,"snapshot-id":7756046587847404627},{"timestamp-ms":1763631128811,"snapshot-id":7836393630613138013},{"timestamp-ms":1763631187018,"snapshot-id":3825740701725564565},{"timestamp-ms":1763631228367,"snapshot-id":8678269831436150991},{"timestamp-ms":1763631248250,"snapshot-id":5296707806511941211},{"timestamp-ms":1763631718360,"snapshot-id":6153398898562664325}],"metadata-log":[{"timestamp-ms":1763554064906,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00000-9b095d6e-dc8c-46ab-a2eb-8c8a0e6babf7.metadata.json"},{"timestamp-ms":1763554064893,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00001-5e523d63-d1d4-4ab2-917e-043425044b33.metadata.json"},{"timestamp-ms":1763554198984,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00002-fafd6564-6e8b-4e8f-8296-4e3247c1401f.metadata.json"},{"timestamp-ms":1763554259988,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00003-06961f84-c43e-4fa7-8b95-56adb6c28e1e.metadata.json"},{"timestamp-ms":1763554279635,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00004-b7c5d99e-a344-4506-8e33-eb17721897ac.metadata.json"},{"timestamp-ms":1763554283523,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00005-32f36809-8b84-45df-9380-f5a929a56e1b.metadata.json"},{"timestamp-ms":1763554473987,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00006-906a269a-3edc-44d9-93dc-361004732510.metadata.json"},{"timestamp-ms":1763556374007,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00007-8093a079-2df4-4081-8d18-d486a7edd986.metadata.json"},{"timestamp-ms":1763631128811,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00008-7fa3943f-7d49-4390-889d-6336251928e5.metadata.json"},{"timestamp-ms":1763631187018,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00009-f995af6a-f978-4c47-950f-d1ff25de1b39.metadata.json"},{"timestamp-ms":1763631228367,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00010-258198be-4f84-44f0-a7c4-34f0889698ac.metadata.json"},{"timestamp-ms":1763631248250,"metadata-file":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/00011-b9e513c3-8cb8-4402-9bf9-342528f6c814.metadata.json"}]}}
INFO:  making POST request to URL http://localhost:5433/api/catalog/v1/oauth/tokens : {grant_type=client_credentials&scope=PRINCIPAL_ROLE:ALL}
INFO:  received response with status code 200, body: {"access_token":"****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************","token_type":"bearer","issued_token_type":"urn:ietf:params:oauth:token-type:access_token","expires_in":3600}
INFO:  making POST request to URL http://localhost:5433/api/catalog/v1/postgres/transactions/commit : {{"table-changes":[{"identifier":{"namespace":["public"],"name":"t_rest_2"},"requirements":[], "updates":[{"action":"add-snapshot","snapshot":{"snapshot-id":7232197677343024400,"parent-snapshot-id":6153398898562664325,"sequence-number":12,"timestamp-ms":1763631731491,"manifest-list":"s3://okalaci/postgres/public/t_rest_2/1480508/metadata/snap-7232197677343024400-1-d2a30889-3b7e-47ee-a09a-b1c01ba2e0b3.avro","summary":{"operation": "append"},"schema-id":2}}, {"action":"set-snapshot-ref", "type":"branch", "ref-name":"main", "snapshot-id":7232197677343024400}]}]}}
INFO:  received response with status code 204, body: <empty>
INSERT 0 1

sfc-gh-mslot

Based on the GUC name I was sort of expecting this would LOG the requests

sfc-gh-mslot · 2025-11-19T14:37:33Z

pg_lake_iceberg/src/http/http_client.c


+	if (EnableHttpClientLogging)
+	{
+		ereport(NOTICE, (errmsg("preparing HTTP request to URL: %s\nRequest Type: %s \nRequest Body: %s", url,


Minor, would have a slight preference for single line where possible and skipping the N/A

making GET request to URL https://...
making POST request URL https://.. : {...}

sfc-gh-okalaci · 2025-11-19T16:44:51Z

Based on the GUC name I was sort of expecting this would LOG the requests

I think log would be too much?

Based on the GUC name I was sort of expecting this would LOG the requests

I guess INFO is the right level to use here: INFO | Provides information implicitly requested by the user, e.g., output from VACUUM VERBOSE. | INFO | INFORMATION

I changed the implementation a bit, after reading through your review. I changed it to also show the responses from the server. It is actually more interesting to have the full traffic, so the name reflects that as well

sfc-gh-abozkurt · 2025-11-20T09:13:34Z

We might want to redact access token in the log if it hits into the server log as well.

sfc-gh-abozkurt · 2025-11-20T10:07:53Z

pg_lake_iceberg/src/http/http_client.c

+			appendStringInfo(postDataInfo, " : {%s}", postData);
+		}
+
+		ereport(INFO, (errmsg("making %s request to URL %s%s",


Suggested change

ereport(INFO, (errmsg("making %s request to URL %s%s",

ereport(INFO, (errmsg("making %s request to URL %s, body: %s",

sfc-gh-abozkurt · 2025-11-20T10:08:38Z

pg_lake_iceberg/src/http/http_client.c

+#include <string.h>
+#include <ctype.h>


nit: move top

sfc-gh-abozkurt · 2025-11-20T10:16:20Z

pg_lake_iceberg/src/init.c


+
+	DefineCustomBoolVariable(
+							 "pg_lake_iceberg.http_client_trace_traffic",


nit: Does pg_lake_iceberg.http_client_debug sound clear?

Especially useful for #68

sfc-gh-okalaci requested a review from sfc-gh-abozkurt November 19, 2025 12:28

sfc-gh-okalaci assigned sfc-gh-mslot and unassigned sfc-gh-mslot Nov 19, 2025

sfc-gh-okalaci requested a review from sfc-gh-mslot November 19, 2025 14:11

sfc-gh-mslot reviewed Nov 19, 2025

View reviewed changes

sfc-gh-okalaci force-pushed the onder/log_http_requests branch from 9900d26 to b5404f6 Compare November 19, 2025 16:49

sfc-gh-okalaci changed the title ~~Add a GUC to log HTTP requests~~ Add a GUC to log HTTP traffic Nov 19, 2025

sfc-gh-okalaci force-pushed the onder/log_http_requests branch from b5404f6 to 2a9c1a9 Compare November 20, 2025 07:09

sfc-gh-abozkurt reviewed Nov 20, 2025

View reviewed changes

sfc-gh-abozkurt approved these changes Nov 20, 2025

View reviewed changes

Add a GUC to log HTTP requests

7dae279

Especially useful for #68

sfc-gh-okalaci force-pushed the onder/log_http_requests branch from a4096d8 to 7dae279 Compare November 20, 2025 16:45

sfc-gh-okalaci merged commit 697be05 into main Nov 20, 2025
63 checks passed

sfc-gh-okalaci deleted the onder/log_http_requests branch November 20, 2025 17:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add a GUC to log HTTP traffic #71

Add a GUC to log HTTP traffic #71

Uh oh!

sfc-gh-okalaci commented Nov 19, 2025 •

edited

Loading

Uh oh!

sfc-gh-mslot left a comment

Uh oh!

sfc-gh-mslot Nov 19, 2025

Uh oh!

sfc-gh-okalaci commented Nov 19, 2025 •

edited

Loading

Uh oh!

sfc-gh-abozkurt commented Nov 20, 2025

Uh oh!

sfc-gh-abozkurt Nov 20, 2025

Uh oh!

sfc-gh-abozkurt Nov 20, 2025

Uh oh!

sfc-gh-abozkurt Nov 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	ereport(INFO, (errmsg("making %s request to URL %s%s",
	ereport(INFO, (errmsg("making %s request to URL %s, body: %s",



		DefineCustomBoolVariable(
		"pg_lake_iceberg.http_client_trace_traffic",

		#include <string.h>
		#include <ctype.h>

Add a GUC to log HTTP traffic #71

Add a GUC to log HTTP traffic #71

Uh oh!

Conversation

sfc-gh-okalaci commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfc-gh-mslot left a comment

Choose a reason for hiding this comment

Uh oh!

sfc-gh-mslot Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-okalaci commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfc-gh-abozkurt commented Nov 20, 2025

Uh oh!

sfc-gh-abozkurt Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-abozkurt Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-abozkurt Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sfc-gh-okalaci commented Nov 19, 2025 •

edited

Loading

sfc-gh-okalaci commented Nov 19, 2025 •

edited

Loading