Course Notes: Software Engineering for Data Scientists in Python

Python, data science, & software engineering

Modularity

Modularity in software engineering refers to the practice of breaking down a software system into separate and self-contained modules or components. It is a design principle that promotes the separation of concerns and the organization of code into smaller, independent units that can be developed, tested, and maintained more easily. Modularity helps to improve the design, development, and maintenance of software systems by breaking them down into smaller, cohesive, and reusable components. It enhances code organization, facilitates collaboration, and enhances the overall flexibility and robustness of the software.

Introduction to packages & documentation

Introduction to Packages in Python:

In Python, a package is a way to organize related modules into a hierarchical directory structure. Packages provide a means to create a namespace and avoid naming conflicts, making it easier to manage and distribute Python code. Packages are often used to group modules that belong to a common functionality or are part of a larger project.

Documentation in Python:

Documentation plays a crucial role in software development, allowing developers to communicate the purpose, usage, and functionality of their code. In Python, documentation can be created using various techniques and tools to help users understand how to use and interact with the codebase effectively.

Intro to pip

pip is a package manager for Python that simplifies the process of installing, managing, and distributing Python packages and libraries. It is a command-line tool that comes bundled with Python, starting from version 2.7.9 and 3.4 onwards.

Example:

Copy
pip install numpy

Reading documentation with help()

The help() function is a built-in function in Python that provides interactive help and documentation on objects, modules, functions, classes, and other Python entities. It allows you to access the docstring associated with an object and obtain information about its usage, parameters, and functionality.

Example:

Copy
help(numpy.busday_count)

Here's a brief overview of how to use the help() function:

• Basic Usage: To use the help() function, simply call it and pass the object or topic you want help with as an argument. For example, help(object) or help(topic). Replace "object" or "topic" with the name of the specific object or topic you want to get help on.
• Object Documentation: When you provide an object as an argument to help(), it will display the associated docstring and other relevant information. The docstring is a string that provides a description and usage details for the object. The help text will appear in the console or the integrated development environment (IDE) where you're running Python.
• Interactive Navigation: When the help text is displayed, you can navigate through the information using the arrow keys, page up/down, and other keyboard controls. This allows you to scroll through the content and view different sections, such as the description, parameters, and examples.
• Exiting Help: To exit the help mode and return to the Python prompt or the code editor, you can press the "q" key or the "Ctrl+C" combination. This will exit the help viewer and allow you to continue with your programming tasks.
• Help on Modules and Packages: In addition to objects, you can also use the help() function to get information on modules and packages. Just pass the module or package name as an argument, and it will display details about the module, including its contents, classes, functions, and other relevant information.

Conventions and PEP 8

Conventions, in the context of programming, refer to a set of agreed-upon guidelines and best practices followed by developers to ensure code readability, maintainability, and consistency. Conventions help make code more understandable for both developers and collaborators, and they often focus on aspects such as naming conventions, code formatting, and code organization.

PEP 8, which stands for Python Enhancement Proposal 8, is a style guide for Python code. It provides a set of recommendations and conventions for writing Python code that is easy to read and understand. PEP 8 serves as a widely accepted standard within the Python community, helping to promote consistency across different projects and making it easier for developers to work with unfamiliar codebases.

Here are the key aspects of PEP 8 and common conventions in Python:

• Code Formatting: PEP 8 suggests using four spaces for indentation and discourages the use of tabs. It recommends limiting lines to a maximum of 79 characters to ensure readability. For longer lines, you can break them into multiple lines using parentheses or backslashes. Blank lines should be used to separate sections of code logically.
• Naming Conventions: PEP 8 provides guidelines for naming variables, functions, classes, and modules. It recommends using lowercase letters and underscores for variable and function names (e.g., my_variable, my_function). Class names should use CamelCase (e.g., MyClass). Module names should be lowercase and, if necessary, can use underscores (e.g., my_module.py).
• Comments and Documentation: PEP 8 encourages the use of descriptive comments to explain code functionality and intentions. Comments should be concise and clear. It also promotes the use of docstrings to provide documentation for functions, classes, and modules. Docstrings help generate useful documentation using tools like Sphinx.
• Imports: PEP 8 provides guidelines for importing modules. It suggests using separate lines for each import statement and grouping imports into three sections: standard library imports, third-party library imports, and local imports. Explicit is better than implicit, so it is recommended to import specific names rather than using wildcard imports.
• Code Organization: PEP 8 advises organizing code logically and grouping related functions and classes together. It suggests separating function and class definitions with two blank lines. Constants should be named using uppercase letters (e.g., MY_CONSTANT). It also recommends avoiding unnecessary code duplication and using helper functions when appropriate.
• Error Handling: PEP 8 encourages the use of structured exception handling using try and except blocks. It suggests specifying the exception type explicitly instead of using a bare except. This helps improve code clarity and prevents unintended errors from being caught.

pycodestyle

To use pycodestyle, which is a command-line tool for checking Python code against the PEP 8 style guide, you can follow these steps:

Copy
pip install pycodestyle

Check Python Code: Once pycodestyle is installed, you can use it to check your Python code. Open a terminal or command prompt, navigate to the directory containing your Python files, and run the following command:

Copy
pycodestyle your_file.py

Replace your_file.py with the actual name of your Python file. You can also specify multiple files or directories to check them all at once.

Writing a python module

Writing a first package

In Python, a package is a way to organize related modules together. It allows you to create a hierarchical structure to manage your code, making it easier to organize, reuse, and distribute your Python code.

A package is essentially a directory that contains one or more Python modules and a special file called init.py. The init.py file is required in each package directory to indicate that the directory should be treated as a package.

Here's an example of a simple package structure:

Copy
my_package/
    __init__.py
    module1.py
    module2.py

In the above example, my_package is the package name, and it contains two modules: module1.py and module2.py.

To use modules from a package, you can import them using the package name followed by the module name. For example:

Copy
import my_package.module1
my_package.module1.some_function()

Alternatively, you can use the from keyword to import specific functions or classes from a module within a package:

Copy
from my_package.module1 import some_function
some_function()

Packages can have sub-packages, forming a hierarchical structure. This allows you to create complex and organized project structures. For example:

Copy
my_package/
    __init__.py
    module1.py
    module2.py
    subpackage/
        __init__.py
        module3.py

In the above example, subpackage is a sub-package within my_package, and it contains module3.py. To import module3 from the sub-package, you can use the following syntax:

Copy
from my_package.subpackage import module3
module3.some_function()

Adding Functionality to Packages

Here are a few ways you can import functionality using the init.py file:

1. Importing specific modules: You can import specific modules within the package and make them accessible when importing the package. Here's an example:

Copy
# in __init__.py
from .module1 import some_function
from .module2 import some_class

With the above code in the init.py file, you can import my_package and directly access some_function and some_class in your desired file:

2. Packages can also define the all variable in their init.py file to specify which modules should be imported when using the from my_package import * syntax. This variable contains a list of strings with the names of the modules to be imported.

Copy
# __init__.py
__all__ = ['module1', 'module2']

By including the above code in the init.py file of the my_package package, you can use the from my_package import * syntax to import module1 and module2.

Making your Package Portable

To make your Python package portable, you need to ensure that it can be easily transferred and used on different systems without encountering compatibility issues.

The requirements.txt file

The requirements.txt file is commonly used in Python projects to specify the dependencies required by your package. It allows users to easily install the necessary packages for your project. Here's how you can use the requirements.txt file:

1. Create a requirements.txt file: In your project's root directory, create a plain text file named requirements.txt.
2. Specify dependencies: In the requirements.txt file, list the required dependencies, with each package on a separate line. You can include the package name and optionally specify a version requirement. For example:

Copy
requests==2.26.0
numpy>=1.21.0

In the above example, requests version 2.26.0 and numpy version 1.21.0 or higher are specified as dependencies. 3. Update the file: As you develop your project and add or remove dependencies, update the requirements.txt file accordingly. Make sure to include all the necessary dependencies for your package to work correctly. 4. Install dependencies: To install the dependencies specified in the requirements.txt file, you can use the following command in the terminal or command prompt:

Copy
pip install -r requirements.txt

This command will read the requirements.txt file and install all the listed dependencies. 5. Distribute your package: When you distribute your package, include the requirements.txt file so that users can easily install the required dependencies. This helps ensure that your package functions properly on different systems.

Note that it's recommended to use a virtual environment for your Python projects to isolate the project-specific dependencies. This way, the project's dependencies won't interfere with other Python installations on the system.

The setup.py file

The setup.py file is commonly used in Python projects to define the metadata and configuration for your package. It allows you to specify information such as the package name, version, author, dependencies, and more. The setup.py file is typically used in conjunction with tools like setuptools to package and distribute your Python project.

Here's an overview of how to use the setup.py file:

1. Create a setup.py file: In your project's root directory, create a Python file named setup.py.
2. Import setuptools: At the top of your setup.py file, import the setuptools package:

Copy
from setuptools import setup

3. Define the package metadata: In the setup() function, provide the necessary metadata for your package. Some commonly used arguments include:

• name: The name of your package.
• version: The version of your package.
• author: The author or maintainer of the package.
• description: A brief description of your package.
• packages: A list of packages or modules to be included in your distribution.
• install_requires: A list of dependencies required by your package.

Here's an example of a setup.py file:

Copy
from setuptools import setup

setup(
    name='my_package',
    version='1.0.0',
    author='Your Name',
    description='A description of your package',
    packages=['my_package'],
    install_requires=[
        'requests>=2.26.0',
        'numpy>=1.21.0'
    ]
)

Customize the metadata and dependencies according to your project's requirements.

4. Build and distribute your package: To build a distribution package for your project, run the following command in the terminal or command prompt:

Copy
python setup.py sdist

This command creates a source distribution package, typically a .tar.gz or .zip file, in a dist directory.

5. Install and use your package: To install your package, use pip and provide the path to the distribution package file. For example: pip install dist/my_package-1.0.0.tar.gz

Once installed, you can import and use your package in other Python projects.

Adding classes to a package

To write a class in Python following the PEP 8 style convention, you can adhere to the following guidelines:

1. Class naming: Class names should be written in CamelCase, starting with an uppercase letter. If the class name consists of multiple words, each word should be capitalized. For example:

Copy
class MyClass:
    pass

2. Docstrings: Include a docstring at the beginning of the class to provide a brief description of the class's purpose and its functionality. Use triple quotes (""") for multi-line docstrings. For example:

Copy
class MyClass:
    """A brief description of the class."""

    def __init__(self):
        pass

3. Indentation and whitespace: Use four spaces for indentation and ensure consistent spacing around operators, commas, and parentheses. Leave a blank line between methods and use blank lines sparingly within methods to improve readability. For example:

Copy
class MyClass:
    """A brief description of the class."""

    def __init__(self):
        pass

    def my_method(self, param1, param2):
        result = param1 + param2

        if result > 10:
            return True
        else:
            return False

4. Method and attribute naming: Use lowercase letters for method and attribute names, separating words with underscores. For example:

Copy
class MyClass:
    """A brief description of the class."""

    def __init__(self):
        self.my_attribute = 10

    def my_method(self, param1, param2):
        result = param1 + param2
        return result

5. Method order: Follow a logical order when defining methods within the class. Generally, the init method is placed first, followed by other methods. Group related methods together to improve code organization.
6. Access modifiers: Python doesn't have strict access modifiers like public, private, or protected. By convention, use a single underscore (_) prefix for internal methods or attributes that are intended to be private. For example:

Copy
class MyClass:
    """A brief description of the class."""

    def __init__(self):
        self._my_attribute = 10

    def _internal_method(self):
        pass

    def public_method(self):
        pass

Note that this convention is a guideline, and the attributes and methods with a single underscore are still accessible from outside the class.

Instantiation

To create an instance of a class in Python, you need to call the class as if it were a function. This process is called "instantiation." Here's an example of how to create an instance of a class:

Copy
class MyClass:
    def __init__(self, name):
        self.name = name

    def say_hello(self):
        print(f"Hello, {self.name}!")

# Create an instance of MyClass
my_object = MyClass("Alice")

# Access instance attributes and methods
print(my_object.name)       # Output: "Alice"
my_object.say_hello()       # Output: "Hello, Alice!"

In the above example, we define a class called MyClass with an init method and a say_hello method. The init method is a special method called a constructor, used to initialize the object's attributes.

To create an instance of MyClass, we call the class as a function and pass any required arguments defined in the init method. In this case, we pass the argument "Alice". This creates a new instance of the class and assigns it to the variable my_object.

Once the instance is created, we can access its attributes and call its methods using dot notation. In the example, we access the name attribute (my_object.name) and call the say_hello method (my_object.say_hello()).