Logo
×

How to Concat two Dataframes in Python

Using Pandas dataframe, we can concat two dataframes or series in Python. So let's take a brief introduction to what is Pandas in Python.

Pandas is a library typically used for data analysis and manipulation in Python programming. It is a dataframes in Python.

The pandas dataframe is a tabular data structure with labelled axes, rows, and columns that is two-dimensional, immutable, and heterogeneous. There are three main parts to a pandas dataframe: data, rows, and columns.

In Pandas, we have pandas.concat() command to concatenate dataframes. The dataframes can be concatenated together. The dimension on which you want to concatenate can be chosen.

Parameters of pandas.concat()

We write this command as:

Syntax:

pandas.concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)

Concatenate pandas objects along one axis while allowing set logic to be applied along the other axes as an option.

Pandas data frame offers several methods for quickly combining series, dataframe, and Panel objects.

Let’s understand the parameters one by one:

  • objs: A sequence or mapping of series, dataframe, or Panel objects make up this parameter.

Unless a mapping is given, the values will be chosen, and the sorted keys will be used as the keys argument. Unless they are all None, a ValueError would be generated, and any None objects will be silently deleted.

Let’s understand it by taking an example.

Example

# code goes from here
# import Pandas
import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)
# using concat command to concat dataframes
new_df = pd.concat([df1, df2])
print(new_df)

Output:

How To Concat Two Dataframes In Python

In the above example, we have 2 dataframes, df1 and df2. We can merge both tables using the pd.concat() method.

  • axis: It is to concatenate along {0,1}; default 0.

Let’s understand it by taking an example.

Example

# code goes from here
# import Pandas
import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)


df2 = pd.DataFrame([
   ['Chin Yen', '1234567879', 'USA'],
   ['Mike Pearl', '2152313213', 'Scood'],
   ['Green Field', '4517825469', 'New Start']
], columns=["Name", "Phone no.", "Country"]
)


# using concat command to concat dataframes
new_df = pd.concat([df1, df2], axis=1)
print(new_df)

Output:

How To Concat Two Dataframes In Python

In the above example setting the axis value equal to 1, both data frames would be joined along with the index.

  • Join:  How to handle other axis' indexes (es).

{‘inner’, ‘outer’}, default ‘outer’.

Inner is for intersection, and outer is for a union.

Let’s understand it by taking an example.

Example

# code goes from here
# import Pandas
import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)


df2 = pd.DataFrame([
   ['Chin Yen', '1234567879', 'USA'],
   ['Mike Pearl', '2152313213', 'Scood'],
   ['Green Field', '4517825469', 'New Start']
], columns=["Name", "Phone no.", "Country"]
)
# using concat command to concat dataframes
new_df1 = pd.concat([df1, df2], join='outer')
new_df2 = pd.concat([df1, df2], join='inner')
print(f'''Outer join
{new_df1}


Inner join
{new_df2}
''')

Output:

How To Concat Two Dataframes In Python

In the above example, when the value of the join parameter is equal to the outer, it returns a new DataFrame, having all columns. But when the value of the join parameter is equal to the inner, it returns a new DataFrame, which has columns common in both DataFrame.

  • ignore_index: bool, default is ‘False’

The results' axis will be labelled 0,..., n - 1. This is helpful when concatenating objects with the concatenation axis lacks relevant indexing information. The join still respects the index values on the other axes.

Let’s understand it by taking an example.

Example

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)


new_df1 = pd.concat([df1, df2])


new_df2 = pd.concat([df1, df2], ignore_index=True)


print(f'''with default value
{new_df1}


after setting ignore_index True
{new_df2}

Output:

How To Concat Two Dataframes In Python

In the above example, when the value of ignore_index is equal to false, both DataFrames will be from indexes 0 to the number of elements each DataFrames have. But when the value of ignore_index is true, the number of indexes will be from 0 to the total number of elements.

  • Keys: sequence, default is 'None'.

It should contain tuples if more than one level is passed. Create a hierarchical index with the passed keys at the top. This means adding an identifier in a specific order to the result indexes.

Let’s understand it by taking an example.

Example

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)


new_df = pd.concat([df1, df2], keys=["level 1", "level 2"])
print(new_df)

Output:

How To Concat Two Dataframes In Python

In the above example, we created 2 levels using keys. It created multi indexed hierarchy.

  • Levels: Utilizing certain levels (unique values) to build a MultiIndex If not, it will be assumed from the keys.

The default is ‘None’.

Let’s understand it by taking an example.

Example

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)


data = pd.concat([df1, df2], keys=["level 1", "level 2"], levels=[["level 1", "level 2", "level 3"]])
print(data)


print("\nlevels in table")
print(data.index.levels)

Output:

How To Concat Two Dataframes In Python

In the above example, we have 2 levels used in the table, but there are 3 levels defined using the level parameter.

  • Names: level names for the generated hierarchical index.
    The default is ‘None’.
    Let’s understand it by taking an example.

Example

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)


data = pd.concat([df1, df2], keys=["level 1", "level 2"], names=["level", "index"])
print(data)

Output:

How To Concat Two Dataframes In Python

In the above example, we provided the name of levels and indexes using the name parameter.

  • verify_integrity: bool,default is False.

Verify if there are any duplicates in the newly concatenated axis. Comparatively speaking to the actual data concatenation, this can be highly expensive.

Let’s understand it by taking an example.

Example

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)


data = pd.concat([df1, df2], verify_integrity=True)
print(data)

Output:

ValueError: Indexes have overlapping values: Int64Index([0, 1, 2], dtype='int64')

In the above example, we are getting an error because the values of indexes are repeating. If we set ignore_index to True, we will not get any error.

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"]
)


data = pd.concat([df1, df2], verify_integrity=True, ignore_index=True)
print(data)

Output:

How To Concat Two Dataframes In Python

Here we can see there is no repetition of data. That’s why there is no error.

  • Sort: bool, default is False.

If the join is "outer," sort the non-concatenation axis if it is not already aligned. This has no impact when join=” inner”, which already maintains the order of the non-concatenation axis.

Let’s understand it by taking an example.

Example

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"],
   index=[1, 3, 2]
)
df2 = pd.DataFrame([
   ['Chin Yen', '1234567879', 'USA'],
   ['Mike Pearl', '2152313213', 'Scood'],
   ['Green Field', '4517825469', 'New Start']
], columns=["Name", "Phone no.", "Country"],
   index=[1, 3, 2]
)


data_1 = pd.concat([df1, df2], axis=1)
data_2 = pd.concat([df1, df2], axis=1, sort=True)
print(f''' when sort=False
{data_1}


when sort=True
{data_2}
''')

Output:

How To Concat Two Dataframes In Python

In the above example, we use the sort parameter. When the sort value is False, it only concatenates data frames together, but when we use the sort value is True, it will concatenate data frames after sorting.

  • Copy: bool, Default is True.

If it is False, it does not copy data unnecessarily.

Let’s understand it by taking an example.

Example

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"],


)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"],


)


data = pd.concat([df1, df2])


data[data.index == 1] = data[data.index == 1].replace("Accounts", "IT")


print(f'''content in data
{data}


content in df1
{df1}
content in df2
{df2}
''')

Output:

How To Concat Two Dataframes In Python

In the above example, we have DataFrame named data, made by contacting df1 and df2. When we make data DataFrame, there is no change in df1 or df2. Let’s try it when the value of the copy parameter is False.

import pandas as pd


df1 = pd.DataFrame([
   ['Chin Yen', 'Lab Assistant', 'Lab'],
   ['Mike Pearl', 'Senior Accountant', 'Accounts'],
   ['Green Field', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"],


)
df2 = pd.DataFrame([
   ['Dewane Paul', 'Programmer', 'IT'],
   ['Matts', 'SR. Programmer', 'IT'],
   ['Plank Oto', 'Accountant', 'Accounts']
], columns=["Name", "Designation", "Department"],


)


data = pd.concat([df1, df2], copy=False)


data[data.index == 1] = data[data.index == 1].replace("Accounts", "IT")


print(f'''content in data
{data}


content in df1
{df1}


content in df2
{df2}
''')

Output:

How To Concat Two Dataframes In Python

In the above example, we can see when we make any change in data, and It also changes the values of df1 or df2. As we change the value of the 2nd row (index = 1) from Accounts to IT, It is also changed in df1.

Related Topics

Latest Project Ideas using Python 2024

Python is one of the well-known languages for programming in the contemporary domain of computer science. The demand to adopt Python for IT purposes is steadily increasing because of its...

7 minutes read.

How to create a dictionary in Python?

How to create a dictionary in python Dictionary is a data structure in Python that represents our data in the form of keys and values. Each value in a dictionary can be...

5 minutes read.

Decision Tree in Python

Decision Tree is one of the most essential algorithms in the area of machine learning for classification and regression. But let us first talk about the lifespan of every machine learning...

12 minutes read.

YOLO Python

YOLO means you only look once.It is a technique to perform object detection is called YOLO. It is the algorithmused by the program to identify items in the image. Earlier detection...

4 minutes read.

End Parameter in python

print(): The python print() function prints the program’s output to the output screen. The output can be an integer value, string value or other value. Syntax: print(“hi”) Output: hi will be displayed on the output...

3 minutes read.

Operator Module In Python

Introduction The operator module is used for performing operations using methods rather than utilizing operators in Python code. The operator module provides several methods for performing the operations.  The operator module contains...

7 minutes read.

How to Define a Function in Python?

What is a Function? In programming, a piece of code that executes a specific task or a group of related operations is known as a function. What does Python Functions Do? If you...

6 minutes read.

Application to get live USD/INR rate Using Tkinter in Python

Tkinter: The standard Python technique for building Graphical User Interfaces (GUIs) is Tkinter, which is included in all popular Python distributions. The only framework included in the Python standard library is...

4 minutes read.

Nested Tuple in Python

If you are a Python learner, this page contains all the information that helps you know about nested tuple and how to access it in python. What is a Tuple? Multiple items...

4 minutes read.

Python enumerate() function

Python enumerate() function The enumerate() function in Python takes a collection (e.g. a tuple) and returns it as an enumerate object. Syntax: enumerate(iterable, start=0) Parameter Iterable:  This parameter represents an iterable object start:    This parameter represents a number defining...

1 minute read.

Why learn Python?

All things considered, learning python would be extraordinary, it'll acquaint you with the universe of dynamic programming dialects, if you're somebody who has had a semester of involvement with C. Python...

4 minutes read.

Python Key Error

What is an Error? Errors are nothing but problems in the program which occur in a program code, and this will stop the execution of the program. It is also called...

6 minutes read.

How to Declare a Variable in Python?

The concept of constants and variables is something that we are studying right from our primary classes. We know that constants are the fixed values whereas variables are those whose...

4 minutes read.

Python all() Function

Python all() Function The all() function in Python returns a Boolean value ‘True’ if all the items in iterable are true or if the iterable object is empty, else it returns False. Syntax all(iterable) Parameter Iterable: An iterable object...

1 minute read.

Python Program for Linear Search

Introduction We employ specific algorithms to efficiently carry out our responsibilities, such as searching for an element in a given data structure. These algorithms fall under the category of searching algorithms....

4 minutes read.

API Requests using Python

What is an API? API stands for Application Programming Interface. It is commonly known as API. It provides an environment that helps two or more computer programs to contact each other....

5 minutes read.

Iterate a Dictionary in Python

In this tutorial, we will learn how to Iterate a Dictionary in Python. Dictionary: In Python, a dictionary is an unordered collection of data values that is used to store data values,...

3 minutes read.

Classes & Objects in Python

Classes and Objects are used in most modern programming languages, mainly in Object-oriented Programming languages. What is meant by Object Oriented Programming language? Usage of objects and their components in order to...

7 minutes read.

Data Drop in Python

Introduction You'll understand how to delete a group of rows from a Pandas dataframe in this article.You can read this article on How to Drop Columns in Pandas to find out...

6 minutes read.

Isreal() Python

Python Programming Language Python programming language is one of the most used programming languages, as it is used widely in the field of software and data analysis, web development, etc. It...

4 minutes read.