Many modern Python applications need information from the internet. A weather app retrieves current weather updates, a news application displays current headlines, and an online shopping service may compare product prices from different sources.

Python offers two common ways to collect online data: using APIs (Application Programming Interfaces) and using web scraping. Although both methods retrieve information from websites or online services, they work differently and are used in different situations.

In this lesson, you will learn what APIs and web scraping are, how they work, and the basic Python tools used for each approach.

What is an API?

An API (Application Programming Interface) is a set of rules that allows one application to communicate with another. Instead of reading a web page like a human, your program sends a request to a server, and the server returns data in a structured format, usually JSON.

Many websites provide APIs so developers can access data safely and efficiently. Common examples include weather information, currency exchange rates, maps and location data, sports scores, and movie details.

Using an API is usually the preferred way to access data because it is structured and designed for developers.

Making an API Request

Python commonly uses the requests library to communicate with APIs.

import requests

response = requests.get("https://api.example.com/data")

print(response.status_code)

The get() function sends a request to the server. The status_code shows whether the request was successful. A status code of 200 usually means the request completed successfully.

Working with JSON Data

Most APIs return data in JSON (JavaScript Object Notation) format. Python can easily convert JSON data into dictionaries and lists.

import requests

response = requests.get("https://api.example.com/user")
data = response.json()

print(data)

Once converted, you can access values just like a dictionary.

print(data["name"])

JSON is one of the most commonly used formats for exchanging data on the web.

API Authentication

Some APIs are public, while others require authentication. Authentication is usually handled using an API key or access token.

headers = {
    "Authorization": "Bearer YOUR_API_KEY"
}

The server checks your credentials before providing access to its data. Always keep API keys private and never include them directly in publicly shared code.

What is Web Scraping?

Web scraping is the process of extracting information directly from a web page. Instead of using an official API, your program downloads the page's HTML and collects the information you need.

• A website does not provide an API.
• You need publicly available information.
• You want to collect data for research or analysis.

Before scraping a website, always review its terms of service and respect any restrictions it may have.

Beautiful Soup

Beautiful Soup is one of the most popular Python libraries for web scraping. It helps you read and navigate HTML documents.

from bs4 import BeautifulSoup

html = """
<h1>Welcome</h1>
<p>Learn Python.</p>
"""

soup = BeautifulSoup(html, "html.parser")

print(soup.h1.text)

Output

Welcome

Beautiful Soup makes it easy to locate HTML elements and extract their contents.

Using Requests with Beautiful Soup

In real projects, requests and Beautiful Soup are often combined. The requests library downloads the web page, and Beautiful Soup processes the HTML.

import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com")
soup = BeautifulSoup(response.text, "html.parser")

Selenium

Some websites load content with JavaScript after the page opens. In these cases, Beautiful Soup alone may not be enough. Selenium is a Python library that automates a real web browser.

• Open web pages
• Click buttons
• Fill out forms
• Scroll through pages
• Extract dynamically loaded content

Selenium is especially useful for testing websites and working with pages that require user interaction.

API vs Web Scraping

Although both methods retrieve online information, they have different purposes.

Whenever an official API is available, it is generally the better choice.

Real-World Applications

• Weather dashboards
• Stock market trackers
• News aggregators
• Travel booking systems
• Price comparison websites
• Social media analytics
• Job search platforms
• Data analysis projects

These technologies allow developers to build applications that use real-time online information.

Best Practices

• Read the API documentation carefully.
• Handle errors such as failed requests.
• Store API keys securely.
• Respect website terms of service.
• Avoid sending too many requests in a short time.
• Use web scraping only on publicly accessible data and in accordance with applicable rules and permissions.

Following these practices helps you build reliable and responsible applications.

Key Takeaways

• APIs allow applications to communicate and exchange structured data.
• Python commonly uses the requests library to work with APIs.
• Most APIs return data in JSON format, which Python can easily process.
• Some APIs require authentication using an API key or access token.
• Web scraping extracts information directly from HTML pages.
• Beautiful Soup helps parse and navigate HTML content.
• Selenium automates browser interactions for websites that rely on JavaScript.
• APIs are generally preferred because they are faster, more stable, and officially supported.
• Web scraping is useful when an API is unavailable, provided it is done responsibly and within the website's terms.
• APIs and web scraping are widely used to build applications that work with live online data.