diff --git a/docs/_quarto.yml b/docs/_quarto.yml index 55417a9..5d97301 100644 --- a/docs/_quarto.yml +++ b/docs/_quarto.yml @@ -159,6 +159,7 @@ book: - href: notes/fetching-data/xml.qmd - href: notes/fetching-data/html-web-scraping.qmd #text: "HTML Data (Web Scraping)" + - href: notes/fetching-data/apis.qmd #- part: "APIs and Integrations" # chapters: diff --git a/docs/images/alphavantage-api-docs-analytics.png b/docs/images/alphavantage-api-docs-analytics.png new file mode 100644 index 0000000..82be5b4 Binary files /dev/null and b/docs/images/alphavantage-api-docs-analytics.png differ diff --git a/docs/images/alphavantage-api-docs-software.png b/docs/images/alphavantage-api-docs-software.png new file mode 100644 index 0000000..115b273 Binary files /dev/null and b/docs/images/alphavantage-api-docs-software.png differ diff --git a/docs/images/alphavantage-api-endpoints.gif b/docs/images/alphavantage-api-endpoints.gif new file mode 100644 index 0000000..05cf9fd Binary files /dev/null and b/docs/images/alphavantage-api-endpoints.gif differ diff --git a/docs/images/alphavantage-api-key.png b/docs/images/alphavantage-api-key.png new file mode 100644 index 0000000..c152b3b Binary files /dev/null and b/docs/images/alphavantage-api-key.png differ diff --git a/docs/images/alphavantage-inflation-data-csv.png b/docs/images/alphavantage-inflation-data-csv.png new file mode 100644 index 0000000..545cc53 Binary files /dev/null and b/docs/images/alphavantage-inflation-data-csv.png differ diff --git a/docs/images/alphavantage-inflation-data.png b/docs/images/alphavantage-inflation-data.png new file mode 100644 index 0000000..1b6b1f8 Binary files /dev/null and b/docs/images/alphavantage-inflation-data.png differ diff --git a/docs/images/alphavantage-inflation-docs-with-url.png b/docs/images/alphavantage-inflation-docs-with-url.png new file mode 100644 index 0000000..a7a2c1f Binary files /dev/null and b/docs/images/alphavantage-inflation-docs-with-url.png differ diff --git a/docs/images/alphavantage-inflation-docs.png b/docs/images/alphavantage-inflation-docs.png new file mode 100644 index 0000000..3a40d5a Binary files /dev/null and b/docs/images/alphavantage-inflation-docs.png differ diff --git a/docs/images/alphavantage-inflation-params.png b/docs/images/alphavantage-inflation-params.png new file mode 100644 index 0000000..39ea485 Binary files /dev/null and b/docs/images/alphavantage-inflation-params.png differ diff --git a/docs/images/alphavantage-inflation-url.png b/docs/images/alphavantage-inflation-url.png new file mode 100644 index 0000000..1096bc3 Binary files /dev/null and b/docs/images/alphavantage-inflation-url.png differ diff --git a/docs/notes/fetching-data/apis.qmd b/docs/notes/fetching-data/apis.qmd index 762e991..995f706 100644 --- a/docs/notes/fetching-data/apis.qmd +++ b/docs/notes/fetching-data/apis.qmd @@ -2,6 +2,8 @@ format: html: code-fold: false + toc-depth: 5 + toc-expand: true jupyter: python3 execute: cache: true # re-render only when source changes @@ -10,105 +12,132 @@ execute: # Application Programming Interfaces (APIs) -We've seen how we can fetch some static data from the Internet, but it's much more helpful to fetch some dynamic data, such as real time market data. +So far, when fetching JSON and CSV formatted data from the Internet, we've primarily explored how to fetch static data files. However accessing dynamic data, such as real-time market information, can be far more valuable. -In order to do this, we'll leverage the capabilities of a programmatic data source known as an **Application Programming Interface (API)**. When we fetch data from an API, the API is the remote server that we're making requests to. We can make a request for some data and the API will give us that data in response. +To access dynamic data, we'll leverage the capabilities of programmatic data sources known as **Application Programming Interfaces (APIs)**. APIs act as the bridge between different systems, enabling them to communicate and exchange data. In practice, we request some data from an API, and the API server provides the data in response. -There are a variety of APIs hosted by a variety of platforms. For example, the New York Times, many social media platforms, and the AlphaVantage API, which is a source of financial data. +APIs allow us to integrate a variety of data into our software applications, including weather updates, stock prices, and social media feeds. A great variety of public APIs exist, but here are just a few notable examples: -Example APIs: + + [New York Times APIs](https://developer.nytimes.com/apis){target="blank"} + + [Google APIs](https://developers.google.com/apis-explorer/#p/){target="blank"} (Maps, Sheets, Drive, Gmail, Calendar, YouTube, etc.) + + [Yelp API](https://docs.developer.yelp.com/docs/fusion-intro){target="blank"} + + [AlphaVantage API](https://www.alphavantage.co/){target="blank"} + + [Spotify API](https://developer.spotify.com/documentation/web-api){target="blank"} + + [CocktailDB Drinks API](https://www.thecocktaildb.com/api.php){target="blank"} + + [MovieDB API](https://developer.themoviedb.org/reference/intro/getting-started){target="blank"} + + [Edamam Food and Nutrition API](https://www.edamam.com/){target="blank"} + + [Pokemon API](https://pokeapi.co/){target="blank"} + + ~~[Twitter APIs](https://dev.twitter.com/rest/public){target="blank"}~~ + + [Facebook Social Graph API](https://developers.facebook.com/docs/graph-api){target="blank"} + + [Instagram API](https://developers.facebook.com/docs/instagram-platform){target="blank"} + + [GitHub API](https://developer.github.com/v3/){target="blank"} + + [US Federal Elections Commission (FEC) API](https://api.open.fec.gov/developers){target="blank"} + + [US Securities and Exchange Commission EDGAR API](https://sec-api.io/){target="blank"} + + [Weather.gov API](https://www.weather.gov/documentation/services-web-api){target="blank"} + + [WMATA APIs](https://developer.wmata.com/apis){target="blank"} - + [New York Times APIs](https://developer.nytimes.com/apis) - + [Google APIs](https://developers.google.com/apis-explorer/#p/) - + Google Maps API - + Google Sheets API - + Google Drive API - + Gmail API - + YouTube API - + etc. - + [Yelp API](https://docs.developer.yelp.com/docs/fusion-intro) -Social Media APIs: - + ~~[Twitter APIs](https://dev.twitter.com/rest/public)~~ (RIP) - + [Facebook Social Graph API](https://developers.facebook.com/docs/graph-api) - + [Instagram API](https://developers.facebook.com/docs/instagram-platform) - + [GitHub API](https://developer.github.com/v3/) -Government and Financial APIs: +To get familiar with APIs, we're going to focus on one specific API, the AlphaVantage API, in more detail. - + [US Federal Elections Commission (FEC) API](https://api.open.fec.gov/developers) - + [US Securities and Exchange Commission EDGAR API](https://sec-api.io/) - + [Weather.gov API](https://www.weather.gov/documentation/services-web-api) - + [AlphaVantage API](https://www.alphavantage.co/documentation/) +## API Documentation -For fun: +To learn how to work with any given API, we must first consult its documentation, to see what capabilities it provides, and what instructions it gives us for how we can request data. Let's go take a look at the documentation for the [AlphaVantage API](https://www.alphavantage.co/){target="blank"}. The AlphaVantage API provides real time historical stock market data and other economic indicators as well. - + [Spotify API](https://developer.spotify.com/documentation/web-api) - + [CocktailDB Drinks API](https://www.thecocktaildb.com/api.php) - + [MovieDB API](https://developer.themoviedb.org/reference/intro/getting-started) - + [Edamam Food and Nutrition API](https://www.edamam.com/) - + [Pokemon API](https://pokeapi.co/) +![The AlphaVantage API documentation site.](../../images/alphavantage-api-docs-software.png){height=350} +## API Keys +In order to use the API, we must first obtain a secret credential known as an **API key**. An API key is like a secret password that we'll use to associate our requests with ourselves. To obtain an API key for this platform, they are instructing us to [fill out a signup form](https://www.alphavantage.co/support/#api-key){target="blank"}. -To learn about APIs, we're going to focus on the AlphaVantage API in more detail. In order to learn how to work with an API, we must first consult its documentation to see what capabilities it provides and see what instructions it gives us for how we can request data from that API. +![Signing up for an AlphaVantage API Key.](../../images/alphavantage-api-key.png){height=350} -## API Documentation +:::{.callout-tip title="Premium Academic Access Keys"} +For students in class, feel free to use one of the "premium" keys that have been shared with you. These keys provide access to all of the data that this API provides. +::: + +### Benefits of API Keys + +API keys are used for **authentication** purposes, to determine who you are, and **authorization** purposes, to determine what you are allowed to do. + +:::{.callout-note title="Example of API Keys for Authorization (Instagram)"} +Let's discuss API keys within a specific example of the Instagram API. + +We know that when we use the Instagram app on our phone, we only have access to see content from users who have public accounts, or private accounts who have authorized us to "follow" them. + +If we're using the Instagram app, and trying to access the photos of a user with a private account who has not authorized us to "follow" them, we won't be able to see their photos. So why would we be able to see those photos if we're asking for that data in a programmatic way? + +The Instagram API needs a way of implementing access controls. This is the role of an API key. If we supply an API key along with our request to the Instagram API. We're saying, "Hey, it's us who's requesting this data". And if we're supposed to have access, show us the photos. Otherwise the Instagram API can deny us access. +::: + + +API Keys also allow the API provider to enforce **rate limits**, which help control the number of requests a client can make to an API within a specific time frame. By associating each API key with a specific client, the server can monitor and limit a given client's request rate, ensuring that no single client can overwhelm the API with excessive traffic. This is particularly useful for maintaining service stability and preventing abuse, as it allows the API provider to throttle high-frequency users or suspicious activity. Rate limiting through API keys not only protects resources from overuse but also ensures fair usage across all clients, helping to maintain performance and availability for legitimate users. -Let's go take a look at the documentation for the AlphaVantage API. The AlphaVantage API provides real time historical stock market data and other economic indicators as well. -![]() -## API Keys -In order to use the API, we must first obtain a secret credential known as an API key. An API key is like a secret password that we'll use to associate our requests with ourselves. Let's talk about API keys within a specific example of the Instagram API. We know that when we use the Instagram app on our phone, we only have access to see videos and photos from users that we follow who have public accounts or who have authorized us to see their photos. -If we're using the Instagram app and we don't have access to see someone's photos, we won't be able to see them. So why would we be able to see those photos if we're asking for that data in a programmatic way? This is the role of an API key. Imagine that if we supply An API key along with our request to the Instagram API. -We're saying, "Hey, it's us who's requesting this data". And if we're supposed to have access, show us the photos. Otherwise the Instagram API can deny us access. And that's one benefit or use case for an API key. API keys are also used for security purposes to protect the API. One technique that hackers use to overwhelm a server is making too many requests in too short of a period of a time. -This may overwhelm the server and cause it to crash, thus denying service to people who want to use that platform. This is known as a dedicated denial of service (DDOS) attack. APIs can prevent a dedicated denial of service attack by implementing what's known as a **rate limit**. When we specify our API key to make a request to the API, if we've made too many requests in a short period of time, the API can deny our request and tell us to wait a minute before making another request so we don't overwhelm their server. -So those are the benefits of API keys and why we might use them. -In order to obtain an API key for this platform, you would have to fill out this form, but I partnered with this API and they've provided me with API keys that I can share with you, and you'll find those in Canvas. The Alpha Vantage API keys that I share with you will give you access to all of the data that this API provides. ## API Endpoints -Once we have an API key, we'll continue to consult the documentation for the AlphaVantage API. With any API, it might provide us with different datasets. Each dataset is provided at a specific URL, known as an **endpoint**. We make a request to a specific endpoint to get a specific dataset, and a different endpoint to get a different dataset. +Once we have an API key, we'll continue to consult the API documentation. A single API might provide multiple different datasets. Each dataset is provided at a specific URL, known as an **endpoint**. We make a request to a specific endpoint to get a specific dataset, and a different endpoint to get a different dataset. -Here in the documentation for the AlphaVantage API, we see a number of different endpoints in the left sidebar. +In the documentation for the AlphaVantage API, we see a number of different endpoints in the left sidebar. -![]() +![AlphaVantage API endpoints.](../../images/alphavantage-api-endpoints.gif){height=350} -This API is a wealth of financial information, including endpoints related to stock data on a daily or weekly or monthly basis, news and sentiment data about a given company, fundamental data about companies, including income statements and balance sheets. Foreign exchange data, cryptocurrency prices and more. +This API is a wealth of financial information, including endpoints related to stock data (on a daily or weekly or monthly basis), fundamental data (including income statements and balance sheets), foreign exchange data, cryptocurrency prices, and more. ## Requesting Data from APIs Now that we have an API key, we can use it to make a request for data from the API. In order to make a request, we can consult the list of endpoints, and consult the documentation for the endpoint that we're interested in. -Let's take the inflation endpoint, for example. When we consult the documentation for the inflation endpoint, we first see a description of the data source where this data comes from, which might be helpful context. We also see parameters or opportunities to customize our request, which will return to talking about in a few minutes. - -But most importantly, we see an example URL that we can use to request the data. Remember, when we make a request for data on the Internet, we're specifying some URL that has the data we want. Let's get started by viewing this URL in the browser. That's the simplest way that we can request this data. +Let's take the [inflation endpoint](https://www.alphavantage.co/documentation/#inflation){target="blank"}, for example. When we consult the documentation for the inflation endpoint, we first see a description of the data source where this data comes from, which might be helpful context. We also see parameters or opportunities to customize our request, which will return to talking about in a moment. -![]() +But most importantly, we see an [example URL](https://www.alphavantage.co/query?function=INFLATION&apikey=demo){target="blank"} that we can use to request the data. Remember, when we make a request for data on the Internet, we're specifying some URL that has the data we want. Let's get started by viewing this URL in the browser. That's the simplest way that we can request this data. +![Inflation data in JSON format](../../images/alphavantage-inflation-data.png){height=350} -We see that by default, the server responds with some JSON formatted data, which is essentially a mixture of lists and dictionaries, some nested structure that you should already be familiar working with. There are ways to customize this request further. Let's take a look at the URL. In this URL, we see that we are providing a specific demo API key. -Which will work once or twice, but then will be kicked out and prevented from requesting more data. So before we move on, we should make this request using one of our API keys. I will quickly change the demo key to one of our keys. After changing the API key, we see we're still able to access the data, but now we can continue exploring the capabilities of this API and other ways to customize our request for this inflation endpoint. +We see that by default, the server responds with some JSON formatted data, which is essentially a mixture of lists and dictionaries, some nested structure that you should already be familiar working with. ### URL Parameters + +There are ways to customize this request further. Let's take a look at the URL. + + +URL: `https://www.alphavantage.co/query?function=INFLATION&apikey=demo` + +In this URL, we see that we are providing a specific demo API key (`apikey=demo`). The demo key might work once or twice, but afterwards we will likely be prevented from requesting more data, with an error message like "The demo API key is for demo purposes only". + +So before we move on, we should make this request using our own API key instead. + +URL: `https://www.alphavantage.co/query?function=INFLATION&apikey=YOUR_KEY` + +After changing the demo key to our key, we see we're still able to access the data, but now we can continue exploring the capabilities of this API and other ways to customize our request for this inflation endpoint. + Let's take a look back at the documentation for this inflation endpoint. +![Inflation endpoint documentation.](../../images/alphavantage-inflation-params.png) + We see there are some additional parameters that we can specify in the URL. For example, a parameter called `datatype`, which will allow us to request this data in a different format. By default, The datatype was JSON, leading to the data that we just saw. But alternatively, especially for those of you who are more familiar and comfortable with spreadsheet style data, we can ask for this data in CSV format, which can be imported into spreadsheet software. -Let's give this a try. All we need to do is add `&datatype=csv` to the end of our URL. -We introduce a new URL parameter to the URL starting with an ampersand (`&`) sign. Now we give the name of the parameter, in this case `datatype`, followed by the equals sign (`=`), and the value of `csv` as instructed by the documentation. When we press enter to make this request, you'll see the browser flicker, and a CSV file will get downloaded into your local file system. +Let's give this a try. All we need to do is add `&datatype=csv` to the end of our URL. We introduce a new URL parameter to the URL starting with an ampersand (`&`) sign. Now we give the name of the parameter, in this case `datatype`, followed by the equals sign (`=`), and the value of `csv` as instructed by the documentation. + +URL: `https://www.alphavantage.co/query?function=INFLATION&apikey=YOUR_KEY&datatype=csv` + +When we press enter to make this request in the browser, we'll see the browser flicker, and a CSV file will get downloaded into the local file system. We can then import the data into spreadsheet software to verify the contents. + +![Inflation data in CSV format.](../../images/alphavantage-inflation-data-csv.png){height=350} + +Here we see the data in CSV format. If you like this format better, you can always use the API to collect some CSV formatted data and upload the files into spreadsheet software. -All right. Let's open the data in spreadsheet software to verify. -Here we see the data in CSV format, and if you like this format better, you can use the API to collect some CSV formatted data and upload them into spreadsheet software yourself. We will return to use many additional endpoints from the AlphaVantage API, but hopefully by now you have a better sense of what it's like to use an API. +We will return to use many additional endpoints from the AlphaVantage API, but hopefully by now you have a better sense of what it's like to use an API.