Python & SQL ft. Azure, DataGrip

For a start, many different databases are available, and they can be classified based on their scale or type of data they store or where they are hosted. In this example we are looking at a Relational SQL-based Database called PostgreSQL, and how to work with it using Python. Examples of similar SQL db are MariaDB, MySQL, and Microsoft SQL.

What is PostgreSQL

PostgreSQL (often shortened to "Postgres") is an open-source relational database management system (RDBMS) that is widely used in enterprise-level applications. It was first released in 1989 as a project of the University of California, Berkeley, and has since evolved into a powerful and robust database platform.

PostgreSQL supports a variety of data types, including text, numeric, boolean, date/time, and many others. It also offers advanced features such as support for transactions, an advanced indexing and querying capabilities, and support for complex data structures such as arrays and JSON.

Learn more about PostgreSQL here

What is PsyCopg2

Psycopg2 is a Python library that provides a Python interface to PostgreSQL databases. It allows developers to interact with a PostgreSQL database from within a Python application.

Psycopg2 provides a wide range of functionality, including support for executing SQL commands, fetching data from a database, and inserting data into a database. It also supports transactions, which allow multiple database operations to be executed as a single atomic unit.

One of the advantages of using Psycopg2 is its support for PostgreSQL's advanced features, such as support for custom data types and indexing methods. It also offers support for connection pooling, which can improve performance by reusing database connections rather than establishing a new connection for each database operation.

Learn more about psycopg2 here

Install psycopg2

pip install psycopg2

Let's begin...

This article is not a SQL or DBMS tutorial, (here is one if you want to learn that ), those topics are prerequisites, we will begin the article by deploying a single instance RDBMS PostgreSQL server on Azure using Azure Databases (More about Azure Databases here), post which we will connect to Database, make tables, and perform queries.

Deployment Diagram

A PostgreSQL Db will be deployed in a Linux-based VM for Azure Databases, which can be connected to using an SSL tunnel by a python script running on a local device (or some web app maybe)

Component Diagram

Step 1 Azure Deployment

1. Create a resource -> PostgreSQL

1. Choose Azure Database for PostgreSQL & Create

1. Create a Single Server for Azure Database for PostgreSQL

1. Enter the necessary details and remember the username & password

2. Choose Configuration {This article used Basic 1vCore 5 GB}

1. Wait for Deployment to Complete

Step 2 Network Config & Connecting to Db

You can use tools like PostgreSQL Admin, Datagrip or Table plus, to Connect to the remote DB and test the connection. These tools can also help to create data or queries in a GUI, but this article doesn't concern with that. Let's just test the connection strings.

1. Open Connection Strings, to expose connection strings for all different tools and languages, we will be utilizing JDBC (For Datagrip) and Python for the script.

1. Open Connection Security

   1. Allow Access to Azure Services

   2. Add Current IP (For testing)

   3. Add other IP that you wish Firewall to allow

   4. Enable Enforced SSL Connection (We will configure this in python script too)

1. Use Plus Sign in Data Sources to Add a PostgreSQL connection in DataGrip, past the JDBC link in URL, enter the Database and Password (saved forever), and test the connection.

Apply settings and click ok

Step3 Python Script

1. Install psycopg2 pip install psycopg2

2. Save SQL Password as env variable

3. Connect to Database, using Python Connection Strings (Observe the SSL mode is required)

   ```python import psycopg2 import os ''' dbname='{your_database}' user='admin123@postgrestestdemo' host='postgrestestdemo.postgres.database.azure.com' password='{your_password}' port='5432' sslmode='true' '''

   #get the connection parameter from os sql_host = "postgrestestdemo.postgres.database.azure.com" sql_user = "admin123@postgrestestdemo" sql_password = os.environ.get('SQL_PASSWORD') sql_db = "postgres"

   #connect to the database try: conn = psycopg2.connect(host=sql_host, user=sql_user, password=sql_password, dbname=sql_db, sslmode='require', port=5432) except Exception as e: print(e)

#open a cursor to perform database operations cur = conn.cursor()

4. Create Tables with Columns and Primary keys (if you want, or do that later in Datagrip maybe)

    ```python
    #create a table in database which saves email and firstname

    cur.execute("CREATE TABLE IF NOT EXISTS user_first (email varchar(255), firstname varchar(255));")

    #create another table in database which saves email and lastname

    cur.execute("CREATE TABLE IF NOT EXISTS user_last (email varchar(255), lastname varchar(255));")

1. Insert Data with Insert Command

    #insert data into the table
   
    cur.execute("INSERT INTO user_first (email, firstname) VALUES ('test@gmail.com', 'test');")
    conn.commit()
    cur.execute("INSERT INTO user_last (email, lastname) VALUES ('test@gmail.com', 'demo');")
    conn.commit()
   

2. Perform Joining Operations

    #join the two tables and get the result
   
    cur.execute("SELECT user_first.email, user_first.firstname, user_last.lastname FROM user_first INNER JOIN user_last ON user_first.email = user_last.email;")
    conn.commit()
    result = cur.fetchall()
    print(result)
   

3. Query Data vs Fetch All Data

    #query the table and print the result
   
    cur.execute("SELECT * FROM user_first;")
    conn.commit()
   
    #fetch one result
    result = cur.fetchone()
    print(result)
   

   

4. Close Connection

    #close the communication with the PostgreSQL
    cur.close()
    conn.close()
   

   The conclusion of this subsection is SQL Queries can be run exactly the same (as we run them normally in SQL Admin tools )with a slight modification, thanks to psycopg2. Go through the documentation of pyscopg2 for more info.

5. Observe the Tables in a better fashion, using DataGrip

   

Step4 Reconfigure Deployment & Logs

1. Check Server Logs for Debugging & Transactions

   

2. Check Errors and other parameters with the Metrics facility

   

3. Upscaling - Once a Price Tier is selected, you cant jump to the next tier, so for this example say I can only expand to up to 2VCore under Basic Tier. Learn more here

Conclusion

In conclusion, PostgreSQL is a powerful and widely used open-source relational database management system. It supports a wide range of data types and advanced features such as transactions and support for complex data structures. Psycopg2 is a Python library that provides a convenient way to interact with PostgreSQL databases from within a Python application. By deploying a PostgreSQL server on Azure and using Psycopg2, developers can quickly and easily build scalable and reliable database-driven applications. However, it's important to note that proficiency in SQL and database management concepts is a prerequisite for working with databases and using tools like Psycopg2.