Python tokens and character set

In this tutorial, we will understand what are character sets used in python and what is meant by python tokens and we will further discuss the type of tokens being used in python.

As we know, Python is a general-purpose, high-level programming language. It was developed to ease the developers by enabling them to achieve their tasks by writing fewer lines of code, and these codes are known as scripts. These scripts consist of character sets, tokens, and identifiers. In this tutorial, we will focus on character sets and tokens.

Character set

A valid set of characters that is suitable for a programming language in scripting is known as the character set. Here, we are referring to the Python programming language. Thus, the Python character set is a valid set of characters identified by the Python language. We can use these characters while writing a script in Python. Following are the ASCII / Unicode characters that are supported in the python programming language:

• Alphabets: It includesall the upper case (A-Z) and lower case (a-z) alphabets.
• Digits: It includesall the digits ranging from 0-9.
• Special Symbols: It includesall kinds of special symbols like, ” ‘ l; : ! ~ @ # $ % ^ ` & * ( ) _ + – = { } [ ] \ .
• White Spaces: It includes the non-printable characters known aswhite spaces such as tab space, blank space, newline, and carriage return.
• Other: Python supportsall ASCII and UNICODE characters.

Tokens

Now, let’s understand tokens in python.

The smallest individual unit in a python program is termed a token. Tokens cannot be further divided into subunits. With tokens, we can build all the statements and instructions in a program. There are several tokens used in python. Let’s discuss each one below:

1. Keywords

 Keywords are those reserved words that have got some unique meaning or importance in a programming language. One has to keep the restraints on these keywords while writing the code. Keywords can’t be used as the name of the variable, the name of the function, or for any other random purpose. They are reserved for special usage only. The numbers of keywords are 33 in Python. Some of them include: Elif, from, try, False, True, class, break, continue, and, as, assert, while, for, in, raise, except, or, not, if, print, import, etc.

Now, let us understand this through an example.

Example:

# python program to understand keywords used in python
for i in range(1, 20):
	
	# Printing the value of i
	print(i)
	
	# Checking whether the value of i is more than 10 or not
	# if the value of i comes out to be more than 10 
	# then the loop will continue
	# Here, if, continue, else, break,
	# for loop are keywords
	if i > 10:
		continue
	
	# if i is less than 10 then break the loop
	else:
	break

Output:

2. Identifiers: Identifiers are another keyword supported in python language. These are the names provided to any variable, function, class, list, methods, etc. for their recognition. As Python is sensitive toward the case(case-sensitive), certain rules are to be kept in mind before naming any identifier.

Some of the rules are listed below to name an identifier: -

As specified above, Python is a case-sensitive language, Thus, it treats the same word differently. For example – cat and CAT are two different words for python. Thus, the case is an important aspect of naming identifiers.

1. Identifiers can begin with an upper case character (A-Z) or a lower case character (a-z). Even an underscore( _ ) is also permissible here.
   
   Apart from these, It can’t begin with other characters.
2. Digits can also be a part of an identifier but can’t be the first character. Besides the upper case, lower case characters, and underscores. An identifier can also consist of a digit, but can’t begin with a digit.
3. An identifier doesn’t permit any other special characters or whitespaces to be part of an identifier.
4. As keywords are reserved for specific tasks, they can’t be a part of an identifier.

Now, let us understand this through an example.

Example:

# Here, tutorials and examples are the valid identifiers


tutorials = 'Hello world'
examples = "Bonjour"


# Driver code 
print(tutorials)
print(examples)

Output:

3. Literals or Values: The static values or data items that are written in a source code are termed literals. Various types of literal are supported in python.

The Following are the types of literals:

• String Literals: The string literals are those literals that are written as text within quotes - single, double, or triple. For example: “Computer Science and Engineering”, ‘India is a secular country’, etc. The triple quotes can be taken into account to write multi-line strings.
•  Character Literals: Character literal is another type of literal used in python. Here, instead of a string, characters are enclosed within a single or double- quotes.
• Numeric Literals: These are other types of literals used in python. They are written in the form of numbers. The following numerical literals are supported in python:
• Integer Literal: It includes both positive and negative numbers along with 0. It doesn’t include fractional parts. It can also include binary, decimal, octal, hexadecimal literal.
• Float Literal: It includes both positive and negative real numbers. It also includes fractional parts.
• Complex Literal: It includes a+bi numeral; here a represents the real part and b represents the complex part.