improve and document test setup

CONTRIBUTING.md

@@ -0,0 +1,67 @@
+# Contributing to Azure Rust SDK
+
+If you would like to become an active contributor to this project please
+follow the instructions provided in [Microsoft Azure Projects Contribution Guidelines](https://azure.github.io/azure-sdk/general_introduction.html).
+
+## Building and Testing
+
+This project uses the [mock_transport_framework](https://github.com/Azure/azure-sdk-for-rust/blob/main/docs/mock_transport.md)
+from the unofficial azure Rust SDKs. The main idea is to record interactions with external services (Kusto) locally, and replay
+the responses in CI/CD and for normal testing. This is particularly useful for end to end tests.
+
+To execute tests against recorded responses run:
+
+```bash
+cargo test --features=mock_transport_framework
+```
+
+> Using `cargo test` will also work, but omit all tests requiring the mock_transport_framework
+
+To record new transactions, first place a `.env` file (omitted by `.gitignore`) in the repository root
+
+```toml
+AZURE_CLIENT_ID="..."
+AZURE_CLIENT_SECRET="..."
+AZURE_TENANT_ID="..."
+KUSTO_SERVICE_URL="..."
+KUSTO_DATABASE="..."
+```
+
+> The provided service principal needs to be able to `create` and `drop` tables in the specified database
+
+Then execute tests in `RECORD` mode:
+
+```bash
+TESTING_MODE=RECORD cargo test --features=mock_transport_framework
+```
+
+> While all credentials and identifiable urls are stripped from recordings, the used database name, the query
+> and responses are committed to source control. So make sure no sensitive data is contained therein. Care
+> should also be taken to reduce the information about DB internals returned from a query - especially
+> when using control commands.
+
+## Style
+
+We use `fmt` and `clippy` for formatting and linting:
+
+```bash
+cargo fmt
+cargo clippy
+```
+
+## PRs
+
+We welcome contributions. In order to make the PR process efficient, please follow the below checklist:
+
+- **There is an issue open concerning the code added** - (either bug or enhancement).
+  Preferably there is an agreed upon approach in the issue.
+- **PR comment explains the changes done** - (This should be a TL;DR; as the rest of it should be documented in the related issue).
+- **PR is concise** - try and avoid make drastic changes in a single PR. Split it into multiple changes if possible. If you feel a major change is needed, it is ok, but make sure commit history is clear and one of the maintainers can comfortably review both the code and the logic behind the change.
+- **Please provide any related information needed to understand the change** - docs, guidelines, use-case, best practices and so on. Opinions are accepted, but have to be backed up.
+- **Checks should pass** - these including linting with cargo fmt and clippy and running tests.
+
+## Code of Conduct
+
+This project's code of conduct can be found in the
+[CODE_OF_CONDUCT.md file](https://github.com/Azure/azure-sdk-for-rust/blob/master/CODE_OF_CONDUCT.md)
+(v1.4.0 of the http://contributor-covenant.org/ CoC).

README.md

@@ -1,28 +1,31 @@
-# Microsoft Azure Kusto  (Azure Data Explorer) SDK for Rust
+# Microsoft Azure Kusto (Azure Data Explorer) SDK for Rust
+
 [![Rust Build, Test, and Lint](https://github.com/Azure/azure-kusto-rust/actions/workflows/build.yml/badge.svg)](https://github.com/Azure/azure-kusto-rust/actions/workflows/build.yml)
 [![codecov](https://codecov.io/gh/Azure/azure-kusto-rust/branch/main/graph/badge.svg?token=0MDnbSizwx)](https://codecov.io/gh/Azure/azure-kusto-rust)
 
-*This repo is still in the early stages of development, and doesn't yet have an official release - use with caution.*
-
+_This repo is still in the early stages of development, and doesn't yet have an official release - use with caution._
 
 ## Best Practices
+
 See the SDK [best practices guide](https://docs.microsoft.com/azure/data-explorer/kusto/api/netfx/kusto-ingest-best-practices), which though written for the .NET SDK, applies similarly here.
 
 ## Need Support?
+
 - **Have a feature request for SDKs?** Please post it on [User Voice](https://feedback.azure.com/forums/915733-azure-data-explorer) to help us prioritize
 - **Have a technical question?** Ask on [Stack Overflow with tag "azure-data-explorer"](https://stackoverflow.com/questions/tagged/azure-data-explorer)
-- **Need Support?** Every customer with an active Azure subscription has access to [support](https://docs.microsoft.com/en-us/azure/azure-supportability/how-to-create-azure-support-request) with guaranteed response time.  Consider submitting a ticket and get assistance from Microsoft support team
+- **Need Support?** Every customer with an active Azure subscription has access to [support](https://docs.microsoft.com/en-us/azure/azure-supportability/how-to-create-azure-support-request) with guaranteed response time. Consider submitting a ticket and get assistance from Microsoft support team
 - **Found a bug?** Please help us fix it by thoroughly documenting it and [filing an issue](https://github.com/Azure/azure-kusto-python/issues/new).
 
 ## Looking for SDKs for other languages/platforms?
+
 - [Node](https://github.com/azure/azure-kusto-node)
 - [Java](https://github.com/azure/azure-kusto-java)
 - [.NET](https://docs.microsoft.com/en-us/azure/kusto/api/netfx/about-the-sdk)
 - [Go](https://github.com/Azure/azure-kusto-go)
 
 ## Contributing
 
-This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+This project welcomes contributions and suggestions. Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
 the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
 
@@ -34,10 +37,12 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
 contact [[email protected]](mailto:[email protected]) with any additional questions or comments.
 
+If you would like to become an active contributor to this project please follow the instructions provided in [Contributing.md](CONTRIBUTING.md).
+
 ## Trademarks
 
-This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 
-trademarks or logos is subject to and must follow 
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
+trademarks or logos is subject to and must follow
 [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
 Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
 Any use of third-party trademarks or logos are subject to those third-party's policies.

azure-kusto-data/tests/e2e.rs

@@ -1,11 +1,8 @@
 #![cfg(feature = "mock_transport_framework")]
-use dotenv::dotenv;
 mod setup;
 
 #[tokio::test]
 async fn create_query_delete_table() {
-    dotenv().ok();
-
     let (client, database) = setup::create_kusto_client("data_create_query_delete_table")
         .await
         .unwrap();

azure-kusto-data/tests/setup.rs

@@ -4,8 +4,10 @@ use azure_core::Error as CoreError;
 use azure_identity::token_credentials::{ClientSecretCredential, TokenCredentialOptions};
 use azure_kusto_data::client::{KustoClient, KustoClientOptions};
 use chrono::Utc;
+use dotenv::dotenv;
 use oauth2::AccessToken;
 use std::error::Error;
+use std::path::Path;
 use std::sync::Arc;
 
 pub struct DummyCredential {}
@@ -23,70 +25,66 @@ impl TokenCredential for DummyCredential {
 pub async fn create_kusto_client(
     transaction_name: &str,
 ) -> Result<(KustoClient, String), Box<dyn Error + Send + Sync>> {
-    let client_id = (std::env::var(azure_core::mock::TESTING_MODE_KEY).as_deref()
-        == Ok(azure_core::mock::TESTING_MODE_RECORD))
-    .then(get_client_id)
-    .unwrap_or_else(String::new);
+    let db_path = Path::new(&workspace_root().unwrap())
+        .join(format!("test/transactions/{}/_db", transaction_name));
 
-    let client_secret = (std::env::var(azure_core::mock::TESTING_MODE_KEY).as_deref()
-        == Ok(azure_core::mock::TESTING_MODE_RECORD))
-    .then(get_client_secret)
-    .unwrap_or_else(String::new);
+    let (service_url, credential, database): (String, Arc<dyn TokenCredential>, String) =
+        if std::env::var(azure_core::mock::TESTING_MODE_KEY).as_deref()
+            == Ok(azure_core::mock::TESTING_MODE_RECORD)
+        {
+            dotenv().ok();
 
-    let tenant_id = (std::env::var(azure_core::mock::TESTING_MODE_KEY).as_deref()
-        == Ok(azure_core::mock::TESTING_MODE_RECORD))
-    .then(get_tenant_id)
-    .unwrap_or_else(String::new);
+            let client_id =
+                std::env::var("AZURE_CLIENT_ID").expect("Set env variable AZURE_CLIENT_ID first!");
+            let client_secret = std::env::var("AZURE_CLIENT_SECRET")
+                .expect("Set env variable AZURE_CLIENT_SECRET first!");
+            let tenant_id =
+                std::env::var("AZURE_TENANT_ID").expect("Set env variable AZURE_TENANT_ID first!");
+            let service_url = std::env::var("KUSTO_SERVICE_URL")
+                .expect("Set env variable KUSTO_SERVICE_URL first!");
+            let database =
+                std::env::var("KUSTO_DATABASE").expect("Set env variable KUSTO_DATABASE first!");
 
-    let service_url = (std::env::var(azure_core::mock::TESTING_MODE_KEY).as_deref()
-        == Ok(azure_core::mock::TESTING_MODE_RECORD))
-    .then(get_service_url)
-    .unwrap_or_else(String::new);
+            // Wee need to persist the database name as well, since it may change per recording run depending on who
+            // records it, is part of the request, and as such validated against.
+            std::fs::write(db_path, &database).unwrap();
 
-    let database = (std::env::var(azure_core::mock::TESTING_MODE_KEY).as_deref()
-        == Ok(azure_core::mock::TESTING_MODE_RECORD))
-    .then(get_database)
-    // TODO This name is hardcoded in CI/CD - this this will only work until a new run is recorded
-    .unwrap_or_else(|| String::from("ctna"));
+            let credential = Arc::new(ClientSecretCredential::new(
+                tenant_id.to_string(),
+                client_id.to_string(),
+                client_secret.to_string(),
+                TokenCredentialOptions::default(),
+            ));
+            (service_url, credential, database)
+        } else {
+            let credential = Arc::new(DummyCredential {});
+            let database = String::from_utf8_lossy(&std::fs::read(db_path).unwrap()).to_string();
+            (String::new(), credential, database)
+        };
 
     let options = KustoClientOptions::new_with_transaction_name(transaction_name.to_string());
 
-    let credential: Arc<dyn TokenCredential> = if std::env::var(azure_core::mock::TESTING_MODE_KEY)
-        .as_deref()
-        == Ok(azure_core::mock::TESTING_MODE_RECORD)
-    {
-        Arc::new(ClientSecretCredential::new(
-            tenant_id.to_string(),
-            client_id.to_string(),
-            client_secret.to_string(),
-            TokenCredentialOptions::default(),
-        ))
-    } else {
-        Arc::new(DummyCredential {})
-    };
-
     Ok((
         KustoClient::new_with_options(service_url, credential, options).unwrap(),
         database,
     ))
 }
 
-fn get_service_url() -> String {
-    std::env::var("KUSTO_SERVICE_URL").expect("Set env variable KUSTO_SERVICE_URL first!")
-}
-
-fn get_database() -> String {
-    std::env::var("KUSTO_DATABASE").expect("Set env variable KUSTO_DATABASE first!")
-}
-
-fn get_client_id() -> String {
-    std::env::var("AZURE_CLIENT_ID").expect("Set env variable AZURE_CLIENT_ID first!")
-}
-
-fn get_client_secret() -> String {
-    std::env::var("AZURE_CLIENT_SECRET").expect("Set env variable AZURE_CLIENT_SECRET first!")
-}
+/// Run cargo to get the root of the workspace
+fn workspace_root() -> Result<String, Box<dyn std::error::Error>> {
+    let output = std::process::Command::new("cargo")
+        .arg("metadata")
+        .arg("--no-deps")
+        .output()?;
+    let output = String::from_utf8_lossy(&output.stdout);
 
-fn get_tenant_id() -> String {
-    std::env::var("AZURE_TENANT_ID").expect("Set env variable AZURE_TENANT_ID first!")
+    let key = "workspace_root\":\"";
+    let index = output
+        .find(key)
+        .ok_or_else(|| format!("workspace_root key not found in metadata"))?;
+    let value = &output[index + key.len()..];
+    let end = value
+        .find("\"")
+        .ok_or_else(|| format!("workspace_root value was malformed"))?;
+    Ok(value[..end].into())
 }

test/transactions/data_create_query_delete_table/0_response.json

@@ -2,13 +2,13 @@
   "status": 200,
   "headers": {
     "content-type": "application/json; charset=UTF-8",
-    "date": "Sun, 24 Apr 2022 07:43:37 GMT",
+    "date": "Sun, 24 Apr 2022 09:33:24 GMT",
     "server": "Microsoft-HTTPAPI/2.0",
     "strict-transport-security": "max-age=15724800; includeSubDomains",
     "transfer-encoding": "chunked",
     "vary": "Accept-Encoding",
-    "x-ms-activity-id": "4a35efa7-62ff-4a20-a0fd-378deec9f61b",
-    "x-ms-client-request-id": "unspecified;7a0e24d7-6932-4a8e-988d-9c118ee0df83"
+    "x-ms-activity-id": "a7c15d5a-a206-4b5a-81c8-c62658f2f318",
+    "x-ms-client-request-id": "unspecified;c16286b2-0025-4db4-927d-33154edcf6ad"
   },
-  "body": "eyJUYWJsZXMiOlt7IlRhYmxlTmFtZSI6IlRhYmxlXzAiLCJDb2x1bW5zIjpbeyJDb2x1bW5OYW1lIjoiRXh0ZW50SWQiLCJEYXRhVHlwZSI6Ikd1aWQiLCJDb2x1bW5UeXBlIjoiZ3VpZCJ9LHsiQ29sdW1uTmFtZSI6Ik9yaWdpbmFsU2l6ZSIsIkRhdGFUeXBlIjoiRG91YmxlIiwiQ29sdW1uVHlwZSI6InJlYWwifSx7IkNvbHVtbk5hbWUiOiJFeHRlbnRTaXplIiwiRGF0YVR5cGUiOiJEb3VibGUiLCJDb2x1bW5UeXBlIjoicmVhbCJ9LHsiQ29sdW1uTmFtZSI6IkNvbXByZXNzZWRTaXplIiwiRGF0YVR5cGUiOiJEb3VibGUiLCJDb2x1bW5UeXBlIjoicmVhbCJ9LHsiQ29sdW1uTmFtZSI6IkluZGV4U2l6ZSIsIkRhdGFUeXBlIjoiRG91YmxlIiwiQ29sdW1uVHlwZSI6InJlYWwifSx7IkNvbHVtbk5hbWUiOiJSb3dDb3VudCIsIkRhdGFUeXBlIjoiSW50NjQiLCJDb2x1bW5UeXBlIjoibG9uZyJ9XSwiUm93cyI6W1siNWY2OTUxZDctZDZlZS00YjgzLWI4MjQtNzk0OTMzNDcyMzg1IiwyMS4wLDMzNS4wLDEyOC4wLDIwNy4wLDFdXX1dfQ=="
+  "body": "eyJUYWJsZXMiOlt7IlRhYmxlTmFtZSI6IlRhYmxlXzAiLCJDb2x1bW5zIjpbeyJDb2x1bW5OYW1lIjoiRXh0ZW50SWQiLCJEYXRhVHlwZSI6Ikd1aWQiLCJDb2x1bW5UeXBlIjoiZ3VpZCJ9LHsiQ29sdW1uTmFtZSI6Ik9yaWdpbmFsU2l6ZSIsIkRhdGFUeXBlIjoiRG91YmxlIiwiQ29sdW1uVHlwZSI6InJlYWwifSx7IkNvbHVtbk5hbWUiOiJFeHRlbnRTaXplIiwiRGF0YVR5cGUiOiJEb3VibGUiLCJDb2x1bW5UeXBlIjoicmVhbCJ9LHsiQ29sdW1uTmFtZSI6IkNvbXByZXNzZWRTaXplIiwiRGF0YVR5cGUiOiJEb3VibGUiLCJDb2x1bW5UeXBlIjoicmVhbCJ9LHsiQ29sdW1uTmFtZSI6IkluZGV4U2l6ZSIsIkRhdGFUeXBlIjoiRG91YmxlIiwiQ29sdW1uVHlwZSI6InJlYWwifSx7IkNvbHVtbk5hbWUiOiJSb3dDb3VudCIsIkRhdGFUeXBlIjoiSW50NjQiLCJDb2x1bW5UeXBlIjoibG9uZyJ9XSwiUm93cyI6W1siY2ViYmVmMzItNTIxYi00MWViLTlhOWUtYTQ5YmE0M2E4NzQ2IiwyMS4wLDMzNS4wLDEyOC4wLDIwNy4wLDFdXX1dfQ=="
 }