When you begin data analysis, the Pandas DataFrame is an indispensable tool. It serves as the foundation of data analysis for efficiently handling tabular data. Like Excel sheets or database tables, it allows you to organize data and perform various operations with remarkable efficiency. Mastering the Pandas DataFrame greatly affects the speed and quality of your subsequent analyses. Therefore, understanding its basic structure and how to create it is essential.

You should also understand the Pandas Series, which is closely related to the Pandas DataFrame. A Series is a data structure that represents a single “column” of a DataFrame. By learning Series, you can gain a deeper grasp of how a DataFrame works.

This article provides a thorough, beginner-friendly explanation of the basic structure of the Pandas DataFrame and how to create one. By clearly outlining the differences between it and the Pandas Series, you will deepen your understanding of the DataFrame. We will also introduce the fundamental creation patterns using the pd.DataFrame method with easy-to-follow code samples.

▶️ Be sure to consult the official documentation for DataFrame and Series as well:

• pandas DataFrame Documentation
• pandas Series Documentation

【What You Will Learn in This Article】

• The basic structure and role of the Pandas DataFrame
• How to create a DataFrame from various data formats (how to use pd.DataFrame)
• The fundamental differences between Pandas Series and DataFrame

【Related Articles】 We have separate detailed articles on how to inspect data after creating a Pandas DataFrame and how to retrieve or select specific data. Once you understand the basics of the DataFrame in this article, be sure to read the following posts to learn practical operations.

• How to inspect a DataFrame:

  【Vol.4】 head() in pandas – Show DataFrame’s First Rows

  【Vol.5】 Pandas info and describe for data structure & stats

• How to extract/select data:

  【Part 6】【Beginner Friendly】Basic Operations with Pandas loc to Extract Rows or Columns by Label【Illustrated】

  and more

【Personal Experience】When I first started, I didn’t understand the difference between a Series and a DataFrame and passed a 2-D list to Series, which caused an error. I realized that being aware that a Series is 1-D and a DataFrame is 2-D is the key to avoiding this initial stumbling block.

Installing Pandas

If Pandas is not installed, install it with the following command.

pip install pandas

Importing Pandas

To use Pandas in your Python code, first import it. It is commonly imported with the alias pd.

import pandas as pd

Pandas DataFrame Basics: Structure and Creation

What Is a Pandas DataFrame?—Your Powerful Partner in Data Analysis

The Pandas DataFrame is the most fundamental data structure for working with tabular data in Python and is the central pillar of data analysis. Like a spreadsheet or database table, it organizes data and lets you perform a wide variety of operations efficiently.

• Read and reshape data from CSV or Excel files:
  【Vol.2】Getting Started with CSV Files in Google Colab
• Extract and process only the data you need:
  【Part 6】【Beginner Friendly】Basic Operations with Pandas loc to Extract Rows or Columns by Label【Illustrated】
• Perform statistical calculations and aggregations:
  【Vol.5】 Pandas info and describe for data structure & stats
• Perform preprocessing for creating graphs

In other words, the DataFrame plays a role in every step of data analysis. By mastering the DataFrame, you can automate data processing that would take a long time by hand and move on to more advanced analysis. In most data-analysis projects, the DataFrame is the star of the show.

The Basic Structure of a Pandas DataFrame

A DataFrame consists of the following elements.

• Data: The values in the table. They can hold various data types (numbers, strings, etc.).
• Column names (Columns): The labels for each column. Each column can be thought of as a one-dimensional data structure called a “Series.”
• Index: The labels for each row. By default it is a sequence starting at 0, but you can set any values you like.

The following illustration shows the structure of a Pandas DataFrame.

Creating a Pandas DataFrame

You can create a Pandas DataFrame from various data formats using the pd.DataFrame() method. Below are the main creation patterns commonly used in data analysis.

Create from a List of Lists

A common approach is to use a “list of lists” in Python (a list that contains other lists). In this structure, each element of the outer list becomes a row, and the elements of the inner lists become the values of each column. It is typical to specify the column names with the columns argument.

data_list_of_list = [[1, 'Alice', 24],
                     [2, 'Bob', 27],
                     [3, 'Charlie', 22],
                     [4, 'David', 32],
                     [5, 'Eve', 29]]

# Create DataFrame by specifying column names with the columns argument
df_from_list_of_list = pd.DataFrame(data_list_of_list, columns=['ID', 'Name', 'Age'])

print("DataFrame created from list of lists:\n", df_from_list_of_list)

DataFrame created from list of lists:
    ID     Name  Age
0   1    Alice   24
1   2      Bob   27
2   3  Charlie   22
3   4    David   32
4   5      Eve   29

Create from a Dictionary

You can also pass a Python dictionary to pd.DataFrame(). In this case, the dictionary keys become the column names of the DataFrame and the values (lists or NumPy arrays, etc.) become the column data. For many cases, this method is an intuitive way to create a DataFrame.

data_dict = {'ID': [1, 2, 3, 4, 5],
             'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
             'Age': [24, 27, 22, 32, 29]}

# Create DataFrame from dictionary
df = pd.DataFrame(data_dict)

print("\nDataFrame created from dictionary:\n", df)
# Assign to variable df for use in later examples (adjust existing code if necessary)

DataFrame created from dictionary:
    ID     Name  Age
0   1    Alice   24
1   2      Bob   27
2   3  Charlie   22
3   4    David   32
4   5      Eve   29

Create from a NumPy Array

If your data already exists as a NumPy array, you can pass it directly to pd.DataFrame() to generate a DataFrame. As with the list-of-lists approach, you can specify the columns and index arguments.

# Import NumPy
# To use NumPy in your Python code, import it first. It is commonly imported with the alias np.
import numpy as np

# NumPy is a library for fast numerical computation,
# but beginners don’t need to worry about its detailed usage at this point.
# For now, it is enough to know that it is sometimes used when creating a Pandas DataFrame.
# It helps you efficiently handle multidimensional arrays of the same data type.

data_np = np.array([[1, 'Alice', 24],
                    [2, 'Bob', 27],
                    [3, 'Charlie', 22]])
df_from_np = pd.DataFrame(data_np, columns=['ID', 'Name', 'Age'])
print("\nDataFrame created from NumPy array:\n", df_from_np)

DataFrame created from NumPy array:
   ID     Name Age
0  1    Alice  24
1  2      Bob  27
2  3  Charlie  22

Creating and Modifying with a Specified Index

The Pandas DataFrame allows you to freely set or change the row labels, called the index.

The index is a label that uniquely identifies each row in a DataFrame. By default, an integer sequence starting at 0 is assigned automatically, but you can set meaningful values—such as dates or IDs—depending on your data. The index plays an important role when efficiently selecting specific rows or when joining multiple DataFrames.

You can set a particular column as the index by using the set_index() method.

# Use the df created in the previous example (redefine or check it here if necessary)
# For example, use the df created from data_dict:
data_dict_for_index = {'ID': [1, 2, 3, 4, 5],
                       'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
                       'Age': [24, 27, 22, 32, 29]}
df_index_example = pd.DataFrame(data_dict_for_index)  # Rename variable to avoid overwriting the original df

print("DataFrame before setting the index:\n", df_index_example)

# 'ID'Set the 'ID' column as the index
df_with_index = df_index_example.set_index('ID')
print("\n'ID'DataFrame after setting the 'ID' column as the index:\n", df_with_index)

# Update the variable df for later examples if needed
# df = df_with_index # Uncomment if this df will be used later

DataFrame before setting the index:
    ID     Name  Age
0   1    Alice   24
1   2      Bob   27
2   3  Charlie   22
3   4    David   32
4   5      Eve   29

'ID'DataFrame after setting the 'ID' column as the index:
        Name  Age
ID              
1     Alice   24
2       Bob   27
3   Charlie   22
4     David   32
5       Eve   29

Other Creation Methods (Reading from Files, etc.)

Besides the approaches above, you can create a Pandas DataFrame in the following ways.

• Read external files such as CSV or Excel (pd.read_csv(), pd.read_excel(), etc.)
• Create from a dictionary of Pandas Series

For details on reading CSV files, see the article 【Part 2】Google Colab and Drive: Easy CSV Loading and Saving | Beginner Drive Integration Guide.

【Cautions When Creating a DataFrame】

When creating a DataFrame from a list of lists, pay attention to the structure of your data—for example, you will get an error if the inner lists do not all have the same number of elements.

Fully Understanding the Differences Between Pandas Series and DataFrame

What Is a Pandas Series?—Its Role and Relation to DataFrame

A Pandas Series is a one-dimensional, labeled data structure. It resembles a list or a NumPy array, but differs in that it carries an index.

It is easier to grasp its role if you think of a Pandas Series as a single column that makes up a Pandas DataFrame.

【When Should You Use Series?】 A Series is useful, for example, when you want to pull out just one column from a DataFrame for analysis. Of course, you can also use it on its own when you need to handle one-dimensional data.

Extracting a Series from a DataFrame

Selecting a specific column from the DataFrame created in the previous section yields a Pandas Series. Let’s confirm this relationship with code.

# Use the df created in the previous section (for example, the df created from the dictionary)
# df = pd.DataFrame({'ID': [1, 2, 3, 4, 5], 'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'], 'Age': [24, 27, 22, 32, 29]}) # Redefine df here if necessary

print("Original DataFrame:\n", df)

# Extract the 'Name' column
names_series = df['Name']
print("\n'Name'column (retrieved as Series):\n", names_series)
print("Type of retrieved data:", type(names_series))

Original DataFrame:
    ID     Name  Age
0   1    Alice   24
1   2      Bob   27
2   3  Charlie   22
3   4    David   32
4   5      Eve   29

'Name'column (retrieved as Series):
 0      Alice
1        Bob
2    Charlie
3      David
4        Eve
Name: Name, dtype: object
Type of retrieved data:  <class 'pandas.core.series.Series'>

Basic Ways to Create a Series

You can also create a Pandas Series from various data formats using the pd.Series() method. To understand that a DataFrame column is in fact a Series, let’s briefly look at the basic creation patterns. We will cover the details in another article.

# Example: create from a list
data_s_list = [10, 20, 30, 40]
series_from_list = pd.Series(data_s_list)
print("\nSeries created from a list:\n", series_from_list)

# Example: create from a dictionary (convenient when you want to specify the index)
data_s_dict = {'a': 10, 'b': 20, 'c': 30, 'd': 40}
series_from_dict = pd.Series(data_s_dict)
print("\nSeries created from a dictionary:\n", series_from_dict)

Series created from a list:
0    10
1    20
2    30
3    40
dtype: int64

Series created from a dictionary:
a    10
b    20
c    30
d    40
dtype: int64

Key Differences Between Series and DataFrame (Structure, Dimensionality, Usability)

When learning Pandas, it is crucial to clearly understand the differences between Series and DataFrame. The main distinctions are as follows.

1. Dimensionality:

   • Series: One-dimensional data structure
   • DataFrame: Two-dimensional data structure (rows and columns)

2. Structure:

   • Series: Data are arranged in a single column, and each item has an index (label).
   • DataFrame: A table format with rows and columns; each column has a name and each row has an index.

The following figure visually demonstrates this structural difference.



3. Usability:

   • There are differences in the code you write to select and manipulate data. For example, from a DataFrame you can select a specific column (Series) by its name or use .loc and .iloc to select rows or individual elements. From a Series, on the other hand, you select elements by index label or position.
   • Another important point is that using [] returns a Series (1-D), while using [[]] returns a DataFrame (2-D).
   • Detailed explanations of these operations can be found in the articles below:
     • 【Part 6】【Beginner Friendly】Basic Operations with Pandas loc to Extract Rows or Columns by Label【Illustrated】
     • 【Part 7】【Beginner Friendly】Extract Rows and Columns by Index Number with Pandas iloc—How-To Guide

   Understanding these differences makes it clear when you should work with a Series versus a DataFrame and lets you write code more efficiently.

✅ Summary

This article thoroughly explained the basic structure of the Pandas DataFrame, the cornerstone of Pandas-based data analysis, and the fundamental ways to create one from lists and dictionaries using the pd.DataFrame() method.

We also deepened our understanding of the Pandas Series—an element that makes up a DataFrame—and the fundamental differences between Series and DataFrame in terms of dimensionality, structure, and usability.

By solidifying these Pandas fundamentals, you are now ready to progress through the various stages of data analysis.

To truly master the Pandas DataFrame, knowing how to create it is only the first step—the subsequent data manipulation is equally important. Be sure to check the upcoming articles to further level up your data-analysis skills with the Pandas DataFrame.

Google Colab CSV 読み込み＆保存入門|pandas で read_csv と to_csv を徹底解説

Google Colab CSV を読み書きする方法を初心者向けに徹底解説。pandas の read_csv と to_csv を使い、Google Drive と連携してファイルを簡単に扱えます。エラー対処法も紹介。

pythondatalab.com

▲ Back to top

<p>The post 【vol.3】pandas dataframe: Basics, Structure & Creation Guide first appeared on Python Data Lab（Pythonデータラボ）.</p>

When performing data analysis in Python, it’s crucial to understand the overall structure of your data. Use the info() and describe() methods from the pandas library to inspect a DataFrame’s schema and summary statistics. In this article, we’ll walk through beginner-friendly examples showing how to use these methods.

Dataset Used

We’ll convert the following dictionary into a pandas DataFrame:

import pandas as pd

data = {
    "Name": ["Taro", "Hanako", "Jiro", "Mika", "Kenichi", "Keiko", "Sho", "Akane", "Takashi", "Aoi"],
    "Age": [23, 29, 35, 42, 18, 33, 27, 24, 31, 30],
    "Occupation": ["Engineer", "Designer", "Teacher", "Doctor", "Student", "Nurse", "Programmer", "Sales", "Lawyer", "Researcher"],
    "Annual Income (¥)": [4500000, 5500000, 4900000, 7300000, 0, 4000000, 6000000, 3200000, 8000000, 5800000],
    "Location": ["Tokyo", "Osaka", "Nagoya", "Sapporo", "Fukuoka", "Tokyo", "Kobe", "Sendai", "Yokohama", "Chiba"],
    "Years Employed": [2, 4, 10, 15, 1, 5, 3, 1, 12, 8]
}

df = pd.DataFrame(data)

DataFrame output:

df

Inspecting Structure with info()

df.info()

From this output, you can see:

1. Entries: 10 rows indexed 0–9, and 6 columns
2. Data types:
   • Name (object): strings
   • Age (int64): integers
   • Occupation (object): strings
   • Annual Income (¥) (int64): integers
   • Location (object): strings
   • Years Employed (int64): integers
3. Non-null counts: All columns have 10 non-null entries (no missing values)
4. Memory usage: 612 bytes (small dataset)

Note: int64 is 64-bit integer; object covers strings or mixed types; non-null means no missing data.

▶️ For reference, see the official info documentation:
pandas DataFrame info Documentation

Examining Summary Statistics with describe()

df.describe()

The key statistics are:

• count: number of non-missing values
• mean: average
• std: standard deviation
• min / max: minimum and maximum
• 25% / 50% / 75%: quartiles

Including All Columns

By default, describe() shows only numeric columns. Use include='all' to include categorical data:

df.describe(include='all')

Categorical Statistics Explained

• unique: number of distinct values
• top: most frequent value
• freq: frequency of the top value
• NaN: indicates that a statistic (e.g. mean) is not applicable

Note: For example, “Tokyo” appears twice so freq=2. In ties, one value is shown.

Author’s Takeaway

The first time I ran df.describe(), I assumed no issues—but later found missing values in categorical columns that broke my model training. Now I always start with df.info() and follow with df.describe(include='all').

Lesson Learned: Don’t rely on numeric summaries alone—check structure and stats together!

▶️ For reference, see the official describe documentation:
pandas DataFrame describe Documentation

Summary

• Use info() to inspect types, non-null counts, and dimensions
• Use describe() to review numeric statistics
• Use include='all' to include categorical columns
• NaN indicates that a statistic is not applicable

Key Statistic Terms

Next time, we’ll cover loc for label-based row and column selection!

▲ Back to Top

<p>The post 【Vol.5】 Pandas info and describe for data structure & stats first appeared on Python Data Lab（Pythonデータラボ）.</p>

In pandas, the Python data analysis library, the DataFrame is a table-like data structure. In this article, we’ll show you how to create a DataFrame and use the handy head() function to preview your data.

▶️ For reference, see the official head documentation:
pandas DataFrame head Documentation

Importing pandas

import pandas as pd

Creating a DataFrame

data = {
  "Name": ["Taro", "Hanako", "Jiro", "Mika", "Kenichi", "Keiko", "Sho", "Akane", "Takashi", "Aoi"],
  "Age": [23, 29, 35, 42, 18, 33, 27, 24, 31, 30],
  "Occupation": ["Engineer", "Designer", "Teacher", "Doctor", "Student", "Nurse", "Programmer", "Sales", "Lawyer", "Researcher"],
  "Annual Income (¥)": [4500000, 5500000, 4900000, 7300000, 0, 4000000, 6000000, 3200000, 8000000, 5800000],
  "Location": ["Tokyo", "Osaka", "Nagoya", "Sapporo", "Fukuoka", "Tokyo", "Kobe", "Sendai", "Yokohama", "Chiba"],
  "Years Employed": [2, 4, 10, 15, 1, 5, 3, 1, 12, 8]
}
df = pd.DataFrame(data)

df

Using head()

Display the First 5 Rows

df.head()

Display a Custom Number of Rows

df.head(3)  # First 3 rows

View Only Column Names

df.head(0)

Note: Using df.head(0) shows no data rows but displays only the column names. It’s useful for checking your DataFrame’s structure and column order.

[Personal Experience] The first time I used head(0), I thought it was an error because nothing appeared. Later I learned it’s a valid way to see just the headers, and now I always start with it to verify columns.

Combining with print()

print(df.head(3))

This prints rows in a plain text format, which can be hard to read. In Jupyter or Colab, it’s better to rely on df.head() for a nicely formatted table.

 Name   Age Occupation Annual Income (¥) Location Years Employed
0  Taro   23   Engineer            4500000    Tokyo             2
1 Hanako   29   Designer           5500000    Osaka             4
2   Jiro   35   Teacher            4900000    Nagoya           10

Saving the Result as a Variable

first_rows = df.head(3)

Comparing with tail()

df.tail()   # Last 5 rows

df.tail(2)     # Last 2 rows

▶️ For reference, see the official tail documentation:
pandas DataFrame tail Documentation

Summary

The head() method is invaluable for quickly inspecting large datasets. Next time, we’ll cover info() and describe() to understand your data’s structure and summary statistics!

▲ Back to Top

<p>The post 【Vol.4】 head() in pandas – Show DataFrame’s First Rows first appeared on Python Data Lab（Pythonデータラボ）.</p>

When analyzing data in Google Colab, you often need to load CSV files saved in Google Drive or edit and save them. In this article, I’ll explain, step by step with screenshots, how to link Google Colab and Google Drive to easily read and write CSV files.

🔰 For basic usage of Google Colab, please refer to the previous article:
👉
[Vol.1][Beginner] Comparing Python Execution Environments: Differences and Usage of Colab, Anaconda, and Direct Installation

Step 1: How to Mount Google Drive to Google Colab

First, mount your Google Drive in Colab. By running the code below, Colab will be able to access files in your Drive. When prompted “Allow this notebook to access your Google Drive files?”, click “Connect to Google Drive.”

from google.colab import drive
drive.mount('/content/drive')

Step 2: Preparing to Work with CSV Files Using pandas

Next, import the pandas library, which you’ll need to read CSV files.

import pandas as pd

Step 3: Reading a CSV File from Google Drive

Once mounting is complete, you’ll see your Drive contents in the file browser on the left. Hover over the CSV file you want to load (here, height_weight.csv), right-click, and select “Copy path.”

Then use the code below to read the CSV. Paste the path you copied into the parentheses of pd.read_csv. In this example, height_weight.csv contains three columns: name, height (cm), and weight (kg).

[Failure Story] I kept getting “No such file or directory” errors because I mis-typed the path. Copying the path directly from Drive solved the issue.

df = pd.read_csv('/content/drive/MyDrive/height_weight.csv')

Step 4: Display the Loaded DataFrame

df

This will display the entire dataset. If there are many rows, use df.head() to view only the first five rows, which helps keep the output manageable.

Step 5: Save as a New CSV File

To save your edited data as a new CSV (for example, height_weight2.csv), use:

df.to_csv('height_weight2.csv', index=False)

This saves the file in Colab’s temporary directory. You can download it via the file browser or move it manually to Google Drive.
[Personal Experience] I once ended my session without moving the file from the temp directory, losing my work. Now I always specify the Drive path when saving.

Note: By default, files save to /content/, which is wiped after the session ends. To keep files, save directly to Drive, e.g.:
df.to_csv('/content/drive/MyDrive/height_weight2.csv').
[Tip] Always specify your MyDrive path to eliminate the risk of file loss.

Full Code Summary

# 1. Mount Google Drive
from google.colab import drive
drive.mount('/content/drive')

# 2. Import pandas
import pandas as pd

# 3. Read CSV file
df = pd.read_csv('/content/drive/MyDrive/height_weight.csv')

# 4. Display data
df

# 5. Save as new CSV
df.to_csv('height_weight2.csv', index=False)

Conclusion

In this article, we showed how to link Google Colab with Google Drive to read and write CSV files. File handling is a fundamental step in data analysis—give it a try! If you run into issues, don’t worry: take it one step at a time. Consistency is the key to improvement.

✅ Summary

▲ Back to Top

<p>The post Google Colab CSV Tutorial for Beginners: Mount Drive & Use pandas【Vol.2】 first appeared on Python Data Lab（Pythonデータラボ）.</p>

To begin with Python, there are three main methods: “Google Colab,” “Anaconda (Jupyter Notebook),” and “Direct Installation.”

I was unsure which to choose at first, so I’ve summarized this for anyone in the same situation.
[Failure Story] I installed Anaconda, but it was so large that my PC slowed down, so I eventually switched to Colab.

* Note: I’ve also included some beginner-friendly notes.

※1: Even the free Colab plan lets you switch your runtime to GPU, but usage time is limited.
※2: Sessions disconnect automatically after a set time, so it’s not ideal for long-running tasks (extended time is available in Pro).
※3: Geared toward those familiar with Python and system internals. Beginners should start with Colab or Anaconda.

How to Use Google Colab (Step-by-Step)

Step 1: Create a Google Account

To use Google Colab, you need a Google account. If you don’t have one yet, create one here.

[Personal Experience] I set up a dedicated sub-account for learning, and data management became much easier.

Step 2: Access Google Drive

Log in to Google Drive, where your Colab notebooks will be saved.

Step 3: Launch Colab from Google Drive

Click “+ New” → “More” → “Google Colaboratory” in the Google Drive menu.

[Failure Story] I couldn’t find Colab at first, but adding the Chrome extension made it appear.

Step 4: Colab Notebook Interface

When Colab launches, you’ll see the notebook interface. Enter code into cells and run them instantly.

Step 5: Run Your First Python Code

Type 1 + 1 in a cell and press “Shift + Enter” to see 2 below.

[Personal Experience] I still remember the thrill of running “1 + 1” and seeing Python work under my hands.

Step 6: Use for Data Analysis and Learning

Colab is Jupyter-compatible, so you can immediately import and use libraries like pandas and NumPy.

[Failure Story] I forgot to import libraries and got errors—make it a habit to write your import statements first.

🔰 What Are pandas and NumPy? (Beginner Notes)

Two essential libraries for Python data analysis are pandas and NumPy.
Here’s a quick beginner overview of their roles.

• pandas:
  A library for handling tabular data (like Excel).
  You can easily read, manipulate, aggregate, and filter data.
  Example: Load a CSV file and extract only the rows you need.
• NumPy:
  A library specialized in numerical computations, handling vectors and matrices at high speed.
  pandas uses NumPy under the hood, making it foundational for data analysis.
  Example: Perform statistical operations or basic machine learning on large numerical datasets.

In Google Colab, you import them like this:

import pandas as pd
import numpy as np

This assigns the aliases pd and np to each library for convenience.
This aliasing convention is used in most tutorials and books.

If it feels unfamiliar, don’t worry.
It’s enough to know these names for now.
We’ll dive deeper into these libraries in future posts.

✅ Summary

▲ Back to Top

<p>The post 【Vol.1】How to Get Started with Google Colab – A Beginner’s Guide to the Python Execution Environment (with Screenshots) first appeared on Python Data Lab（Pythonデータラボ）.</p>