Dave's PostgreSQL Stuff

DBeaver's Visual Query Builder

Wed, 26 Nov 2025 20:15:00 +0000

Quick! You need to create a complex query, but you don't remember how to join the tables. And you need to get it done ASAP.

You are in luck if you have an enterprise edition of DBeaver handy. You can drag and drop the tables and then adjust the clauses as needed without having to dig into the table definitions.

1. Open the database

The PostgreSQL DVD Rental database.

I am using the PostgreSQL DVDRental dataset for these examples. I have opened the connection and listed the tables.

As you can see, the database contains numerous tables.

We are going to build a query visually to get information about the customers.

2. Open a SQL Console

Open SQL Console

The second step requires an SQL Console. You will find it under the SQL Editor tab.

3. Select Builder

On the right side, Select Builder

On the right side of the console pane, tilted 90 degrees, is a tab labeled Builder.

4. You are ready to drag and drop tables

The Query Builder Is Ready to use.

You have three new panes. The top one is where you drop the tables. The bottom left is where we tailor the query. The query will then appear in the bottom right pane.

Now drag the payment, rental, and customer tables from step 1 separately to the top pane, the one that says 'no active query',

You can rearrange the tables to create a harmonious relationship map that aligns with your artistic aesthetic.

You should have something like this:

The tables are joined and ready for querying.

4. The query

The lower right display will have the query.

select
    *
from
    public.payment p
inner join public.customer c on
    p.customer_id = c.customer_id
inner join public.rental r on
    c.customer_id = r.customer_id
inner join public.payment p on
    r.rental_id = p.rental_id

Not bad, considering you did not have to look up how to join the table.

Refine here

5. Refine the query

Now we get to the lower left window. This is where we refine. Select the Where tab on the left and then the icon with the plus sign on the right.

We want to skip the customers who have not made a payment.

Making sure that payment.amount is greater than zero.

Note that the query is automatically adjusted to match the changes.

And we will switch to the sort tab, add an expression that asks for the payments be listed in descending order.

Add a sort on the amount column, in descending order.

And again, the query is adjusted to match the sorting demand.

6. The result

The query is ready to run.

select
    *
from
    public.payment p1
inner join public.customer c on
    p1.customer_id = c.customer_id
inner join public.rental r on
    c.customer_id = r.customer_id
inner join public.payment p2 on
    r.rental_id = p2.rental_id
where
    p1.amount > 0
order by
    p1.amount desc

And the result. 
The Results

We can eliminate excess data by removing the SELECT * statement and specifying the desired columns.

Specifying specific columns

If you find yourself fighting the syntax of SQL, the Query Builder lets you concentrate on the output and saves you from the frustration.

CSV Loading Error Fix: Data Too Long Or Value Too Long

Thu, 13 Nov 2025 20:19:00 +0000

You are trying to load a CSV file into a database, and almost inevitably, you run into an ERROR: value too long for type character varying 64 if you are using PostgreSQL. Or SQL Error [1406] [22001]: Data truncation: Data too long for column 'Name' at row 1 if you are using MariaDB or MySQL.

It doesn't matter if you are using the DBeaver application or the native database import tools. Newbies will look at the first line of the CSV file to examine the very first line and scratch their heads, as there is no error in the first line.

The database is ingesting the data one line at a time, so each line in the file is the first line. Your line in the CSV file is hiding deep in the file. And there may be many of them in there.

The error message you receive is helpful, but it DOES NOT tell you which line in the CSV file is causing you this grief. You could use an 'eyeball inspection' of the data, which quickly becomes tedious. And if the CSV file has thousands of lines, your odds of finding the offending line grow longer.

Splitting up the file into smaller chunks is labor-intensive.

Example

I was loading data from the HMS Titanic data set when I received the following error:

Error occurred during batch insert
(you can disable batch insert to skip particular rows).

Reason:

SQL Error [22001]: Batch entry 0 INSERT INTO public.titanic ("PassengerId","Survived","Pclass","Name","Sex","Age","SibSp","Parch","Ticket","Fare","Cabin","Embarked")

VALUES (('308'::int8),('1'::int8),('1'::int8),('Penasco y Castellana, Mrs. Victor de Satode (Maria Josefa Perez de Soto y Vallejo)'),('female'),('17'::int8),('1'::int8),('0'::int8),('PC 17758'),('108.9'::real),('C65'),('C')) was aborted: ERROR: value too long for type character varying(64) Call getNextException to see other errors in the batch.

A Difference Between PostgreSQL and MariaDB/MySQL

PostgreSQL will return the offending line in the error message, as shown above. MariaDB and MySQL do not. They provide only SQL Error [1406] [22001]: Data truncation: Data too long for column 'Name' at row 1.

DBeaver's import tool selected a VARCHAR(64) for the name field after examining some of the records. The unfortunate Penasco y Castellana, Mrs. Victor de Satode (Maria Josefa Perez de Soto y Vallejo needs more space. And there may be other records with names as long, or longer.

Solution

You can use Google Sheets to examine the CSV file before attempting to load it into the database.

We can use =max(arrayformula(len(D1:D891))) on the columns with the names to determine the maximum length of a name. The longest name in this data is eighty-two characters long.

If you read the linked blog post, you will see that I defined the Name column as a TEXT field to solve the issue. But what if you are trying to save space? Knowing the width of that column is valuable.

So now you have a tool to determine the width of a column when you get the dreaded 'value too long'

Loading The Titanic Dataset Into PostgreSQL With DBeaver Part 3

Wed, 12 Nov 2025 17:09:00 +0000

In the previous post, we asked AI to make recommendations to help clean up the data loaded directly from a CSV file. The initial data load for the Name failed because a VARCHAR(64) was estimated to be insufficient to hold the data. But it was not, and a TEXT field was used instead.

So now we want to save some disk space!

The first obvious candidate may seem to be the Name field; a TEXT field can hold up to a gigabyte of data. None of the names is that long. If we switch to VARCHAR, we do not gain space, as both have the exact underlying mechanism in PostgreSQL.

PassengerId

There are only 891 records, so we don't have to worry about growth. This is a 'fixed' list, as no one else is going to be added to this list, so we can treat the column as having a fixed width. We can ask for the current maximum value of the columns. In the 'real' world, when creating a table, you want your primary key column to be sized to allow for growth, hopefully enormous growth. However, in cases like this, you do not have to leave room for future growth.

SELECT MAX("PassengerId") as "Passengerid",
       MAX("Survived") as "Survived",
       MAX("Pclass") as "Pclass",
       MAX(LENGTH("Name")) as "Name",
       MAX("Age") as "Age",
       MAX("SibSp") as "SibSp",
       MAX("Parch") as "Parch" ,
       MAX("Fare") as "Fare"
FROM titanic;

We can save space by using a SMALLINT for the PassengerId (reducing it from four bytes to two). And we can shrink other columns too.

ALTER TABLE public.titanic2 ALTER COLUMN "PassengerId" TYPE smallint USING "PassengerId"::smallint;
ALTER TABLE public.titanic2 ALTER COLUMN "Age" TYPE int2 USING "Age"::int2;
ALTER TABLE public.titanic2 ALTER COLUMN "Survived" TYPE boolean USING "Survived"::boolean;
ALTER TABLE public.titanic2 ALTER COLUMN "SibSp" TYPE smallint USING "SibSp"::smallint;
ALTER TABLE public.titanic2 ALTER COLUMN "Parch" TYPE smallint USING "Parch"::smallint;
ALTER TABLE public.titanic2 ALTER COLUMN "Embarked" TYPE varchar(1) USING "Embarked"::varchar(1);

Using pgstattuple

We can use pgstattuple to see the results of this data size diet.

The original 'titanic' table

The revised 'titanic2' table

The length of the live tuples has shrunk from 90,111 to 83,319 bytes. And free space and free percentage have climbed too. A smaller table is loaded into memory faster, scans more quickly, and requires less space in a backup.

Migration From MySQL To PostgreSQL In Five Steps Using DBeaver

Sat, 01 Nov 2025 15:21:00 +0000

I wrote a post in my MySQL blog on migrating from MySQL to PostgreSQL using DBeaver. You can pass it along to your acquaintances who want to get off the Dolphin and on the Elephant.

Not only will DBeaver move your tables and data, but you can compare them afterwards. In the post, I outline the process in five steps. DBeaver will let you do it in four.

Please share this blog with your friends to promote PostgreSQL.

Loading The Titanic Passenger Data Into PostgreSQL With DBeaver Part 2

Wed, 22 Oct 2025 12:08:00 +0000

In the last edition of this blog, the passenger list data from the HMS Titanic was loaded into a preliminary database. Now it is time to refine.

I am using DBeaver Enterprise 25.2.0. PostgreSQL 18, and Github Copilot with Gpt-4.

Prompt: Any recommendations on improving this table for storage efficiency? This prompt was entered into the DBeaver AI Assistant.

To improve the storage efficiency of the "titanic" table, we can consider the following recommendations:

### 1. **Optimize Data Types**

- **"PassengerId"**: If the number of passengers is not expected to exceed 2,147,483,647, the `int4` type is fine. However, if the range is smaller (e.g., less than 32,767), you could use `int2` (smallint) to save space.

- **"Survived"**: Since this is likely a binary value (0 or 1), you can use `boolean` instead of `int4`.

- **"Pclass"**: If the class is limited to 1, 2, or 3, you can use `int2` (smallint) instead of `int4`.

- **"Sex"**: If this column only contains "male" and "female", consider using an `enum` type or a `char(1)` ('M' or 'F') to save space.

- **"Age"**: If precision beyond one decimal place is not required, consider reducing the precision or using `numeric` with a defined scale.

- **"SibSp" and "Parch"**: If these values are small integers (e.g., number of siblings/spouses or parents/children), you can use `int2` (smallint).

- **"Ticket"**: If ticket numbers are of fixed length, consider using `char(n)` instead of `varchar`.

- **"Fare"**: If precision beyond two decimal places is not required, consider using `numeric` with a defined scale (e.g., `numeric(10,2)`).

- **"Cabin"**: If cabin identifiers are of fixed length, consider using `char(n)` instead of `varchar`.

- **"Embarked"**: If this column only contains a small set of values (e.g., 'C', 'Q', 'S'), consider using an `enum` type or `char(1)`.

Comment: This is what I was looking for. Saving space is always good and these are solid recommendations.

### 2. **Normalize the Table**

- If columns like "Sex", "Embarked", or "Pclass" have repetitive values, consider normalizing them into separate lookup tables. For example:

- Create a "sex" table with "id" and "value" columns.

- Create an "embarked" table with "id" and "value" columns.

- Replace the original columns with foreign keys referencing these tables.

Comment: Interesting idea, but will the overhead of an extra lookup to the other tables offset any gains? This suggestion is not getting a 'warm fuzzy' from me.

### 3. **Remove Redundant Columns**

- If any columns are not used frequently or can be derived from other columns, consider removing them. For example, if "Pclass" can be inferred from another column, it may not need to be stored.

Comment: This is intriguing. Dropping redundant columns would help, but how do I make the inference to determine if and how Pclass status?

### 4. **Indexing**

- Add indexes to columns that are frequently queried, such as "PassengerId", "Survived", or "Embarked". However, keep in mind that indexes increase storage usage, so only index columns that are necessary for performance.

Comment: This is a pretty generic suggestion.

### 5. **Partitioning**

- If the table grows significantly, consider partitioning it by a column like "Pclass" or "Embarked" to improve query performance and storage management.

Comment: Hopefully, this dataset is closed. :-) If this was a growing set, or larger, or needed faster lookups, this would be a good consideration.

### 6. **Compression**

- Use PostgreSQL's `TOAST` mechanism for compressing large text fields like "Name", "Ticket", and "Cabin". This is automatically handled by PostgreSQL for large text/varchar fields, but you can ensure it is enabled.

Comment: This is a reach. Nothing in a row of this data will trigger TOAST.

### Example of an Optimized Table Definition

Here’s an example of how the table could look after applying some of these recommendations:

CREATE TABLE titanic (
    "PassengerId" int4 PRIMARY KEY,
    "Survived" boolean,
    "Pclass" int2,
    "Name" text,
    "Sex" char(1),
    "Age" numeric(4,1),
    "SibSp" int2,
    "Parch" int2,
    "Ticket" varchar(20),
    "Fare" numeric(10,2),
    "Cabin" varchar(10),
    "Embarked" char(1)
);

Comment: At first glance, this looks good. But do I want Age as a Numeric
field instead of an Int?

### Additional Notes
- If you normalize the table, you will need to create additional tables for "Sex" and "Embarked" and replace their columns with foreign keys.
- Always test the impact of these changes on your queries and storage before applying them in production.

Conclusion
Github Copilot did what a DBA would do when looking to optimize this data. It looked for columns where data could be saved. It considered partitioning and compression.

AI is a pattern-matching algorithm, and it is unsettling to realize your career was a pattern. 

Next time, we will look at making the changes and see how it looks after the iteration.

Loading The Titanic Passenger Data Into PostgreSQL With DBeaver Part 1

Wed, 08 Oct 2025 14:13:00 +0000

The Sinking of the RMS Titanic in 1912 has shocked and entertained for over a century. I will admit to being shocked to find a GitHub Repo with a CSV formatted file with the passenger list. This file is an interesting glimpse into the lives of the folks who sailed on the ill fated ship so long ago. And interesting datasets are always good for examples. I had been looking for a project to show off what I call 'Lifting and Shifting' but others call ETL.

At the PG NYC 2025 Conference, Ryan Booz had a session title Transforming Data with the Power of PostgreSQL and SQL. For many years, ETL was considered as the way to move data from one source to another Ryan argued that it should be ELT, and rather convincingly.

ETL?

ETL is a three-step data integration process (Extract, Transform, Load) used to collect raw data from various sources, process it into a consistent, usable format, and then load it into a central data warehouse or database for analytics and business intelligence.

Spreadsheet conversions are a fairly common task and DBeaver is a slick way to make the change.

ELT?

The 'T' is often the tricky part. Databases have a lot of tools to manipulate data. So why put off using the power of the database until the end? What if we loaded the data into a database as an intermediate step and used the power of the database to transform the data into a data final form? This seems like an obviously better practice. So how do we do it?

Peek At The Data

Using the above link, I down loaded the CSV file of the passenger list. Using the DBeaver Community Edition, I used File -> Open to access the titanic.csv file. This was done to investigate the file only and does not actually do anything to import the data.

The data can be viewed at this point. A quick review shows nothing out of the ordinary. Foreshadowing - there was something wrong. I like to perform this step just to make sure there is nothing obviously wrong with the data - bad encoding, weird line feeds, sudden changes in character sets, and the like. There maybe formatting issues (JSON, XML, SGML) or other anomalies. I would rather find them at this step than later discovering the data was wrong from the start.

The E

The next step is to import the data to a new PostgreSQL 18 instance. I connected to the instance. Next, Right click on the tables and select Import Data.

And then we need to specify the Import source as a CSV file. DBeaver can use a database table, XML, XLXS file. And yes you can move from Database X to Database Y too,

Now select Input File.

We browse for the titanic.csv file. At this point we can change the importer settings, but the default values were perfectly fine in this case. Often, you have have to change delimiters from single to double quotes, or end of line characters. Life is simpler if you select the proper character set so you do not lose data.

Select Next and then Preview Data.

We can customize the input settings to change the data types or change the column names. again, the defaults are always a good first choice.

The preview looks good, at least at this point.

The settings for loading that data into the database can now be set. I went with the defaults but you may want to bulk load the data, log queries, or even truncate a table from a previous attempt.

And we 'Proceed'. At least until we hit an error.

Okay, so it is time to realize we have an issue. This is fairly common in importing data that one (or more than one) records does not comply with what we thought was the layout. We have a name that can not be used with our varchar(64) specification. At this point, I want ALL the data. So I backed out of the conversion and restarted.

This time I changed the length of the Name column. The heuristics thought that a varchar(64) would suffice but I changed it to a text. PostgreSQL has an amazingly rich set of data types, but your database of choice may not - and you need to check. This is a simple case with one row that does not comply. But often the case is that you have a CSV with many rows that are non-complaint. And the non-complaint rows need to be corrected or the target rows need to be adjusted.

Is using TEXT wasteful? Maybe if I had counted the maximum length of the name column used that number for a varchar things would have been better. The trade off is trying to be perfect on the first conversion versus adjust as needed. Remember this is an intermediate stage and we can trim later, if needed.

Rerun the dialogs and now we have all of our data in a table.

This is the end of part one. We now have the data from a CSV file loaded into a temporary database. Next comes a serious look at the data to see what we can do to save space, add value to the, or improve the usability.

Do we want to keep 'female' and 'male' or using a binary representation. Would it help to have separate first and last name columns? Do we expand the Embarked data. Do we know what all the columns represent? All that will be in Part 2.

PostgreSQL 18 Old & New

Tue, 30 Sep 2025 12:25:00 +0000

Learning Structured Query Language can be frustrating when double-checking that what you wanted to have done is actually what was done. PostgreSQL 18 has 'OLD and NEW support for RETURNING clauses in INSERT, UPDATE, DELETE, and MERGE commands'. Now you can get instant feedback.

The addition of the RETURNING clause in the previous version made MERGE much easier to use. Now it makes other commands easier.

To demonstrate, let's create a table with one column that is designated as a unique, primary key integer and insert a value.

create table foo (a int unique primary key);

insert into foo (a) values (1);

Now is the point where many of us would run a SELECT a FROM foo, just to double check that indeed there is a 1 in column a.

There is now an option in PG 18 to use RETURNING, and it provides the previous 'old' value of a column along with the 'new' value.

insert into foo (a) values (2) returning old.a, new.a;

a|a|
-+-+
|2|

The 'old' value was a NULL, and the new value is 2.

I suggest adding an alias to make things more explicit.

insert into foo (a) values (3) returning old.a as "Old", new.a as "New";

Old|New|

---+---+

| 3|

Upserts

As I mentioned earlier, sometimes doubt creeps into your mind when you enter or update data.

RETURNING also works with UPSERTS (Thanks to the brilliant Magnus Hagander, who pointed this out out in

his presentation at the PG NYC 2025 show and his slides are available here).

In the following example, we have a conflict because there is already a '2' in the table, so the

ON CONFLICT path is taken. Using RETURNING in this case takes away any doubt of what happened when

this query was executed.

insert into foo(a) values (2)
on conflict(a) do update 
  set a = 99
returning old.a as old, new.a as update

old|update|

---+------+

2| 99|

PostgreSQL 18 Release Notes

Sun, 28 Sep 2025 12:58:00 +0000

The PostgreSQL 18 Release Notes are like a great noir novel, full of surprises and intrigue. But we know who did it - the amazing PostgreSQL community.

If you have not perused the Notes, I advise you to do so now. They contain many of what I call 'Wow!' level items, such as the redesigned I/O subsystem, skip scan lookups, virtual generated columns, and more. The new Uuidv7 function and temporal constraints would be a significant leap forward. The OLD and NEW support for RETURNING will help lessen the learning curve for many who are new to SQL.

But go down to the seemingly more mundane items. Removing redundant self-joins, merge-joins can now use incremental sorts, and internally reordering the keys of SELECT DISTINCT to avoid sorting are the types of improvements that are not as attention-getting as the items in the preceding paragraph, but will make life much nicer for us.

Each release of PostgreSQL is faster than the previous one, and early reports report that this has happened again. This is an astounding delivery for those of us with backgrounds in other databases. All too often, new features came with a trade-off in performance that relegated upgrades to the back burner.

Please read the Acknowledgments. This is a list of the good folks who have worked very hard on our behalf to deliver an improved version of PostgreSQL. And they are already working on 19! If you run into any of them, please give them a big 'thank you'.

Book Review - Mastering PostgreSQL 17

Wed, 06 Aug 2025 19:43:00 +0000

Authors find it challenging to write a comprehensive book about PostgreSQL for several reasons. The first is that a new version of PostgreSQL is released every year with changes, new features, and tweaks. I am lucky to have been able to review several new database titles each year, and I would like to let you know that Hans-Jurgen Schonig's Mastering PostgreSQL is a well-written reference.

The first chapter of this book covers changes in 17. The explanations and examples are clear, concise, and easy to comprehend. For instance, the CPY command is convenient, and in 17, it can handle errors. The book's example of this new ability quickly shows how it works—short and sweet!

From there, it covers transactions and locking, indexing, advanced SQL log files, system statistics, optimizing queries (very well done here), how to write stored procedures, security, backups, replication (also very well done), extensions, and troubleshooting. I will have a longer review later this month, covering the outstanding parts of the book that you will need to pay strict attention to..

I highly recommend this to those new to administering PostgreSQL systems or looking to enhance their skills. For instance, the authors start the chapter on replication with a summary of the Write Ahead Log. Understanding the WAL is not only key to transactions but also essential to proper replication. The material on Point In Time Recovery and handling conflicts needs to be studied closely before you have to try it for the first time in real life.

Data Security and AI - Sharing Your PostgreSQL Database With Ollama AI

Fri, 01 Aug 2025 14:45:00 +0000

You probably saw the story about your public ChatGPT queries being indexed by Google and other search engines. Yikes!

You may want AI with your data but do not wish your prompts to end up in a search engine, especially if you are working with sensitive data.

Previously, we covered using some of DBeaver's Artificial Intelligence Assistant features with Ollama. Ollama is popular because it runs on your system, keeping your data from travelling over networks to an AI and back. What happens only on your system stays on your system, to paraphrase the Las Vegas motto. Paranoia in defense of your data is a good thing.

Secure By Default

We will start with an example of preference settings for DBeaver Enterprise Edition for Ollama. DBeaver has other security options I want to bring to your attention when working with an AI. Ollama differs from most other AIs because you do not have to obtain a key to work with it.

AI Configuration - DBeaver Enterprise Edition

The ability to tailor what you send to the AI should be your next step. You can also configure what to share with the AI. The default is to share only the metadata—tables, column names, and Views—but not the data. Optionally, sharing information about keys and object descriptions can improve the AI's job, at the expense of extra token consumption.

Send database structure options in this example, including foreign, unique, and primary keys.

Note that you can send table data to the AI to help it develop a better solution. This is the only option, and you can make the choice. If you do not want to share, you do not have to. The query will also run only on the local machine.

Setting The Scope

The AI Assistant can direct the scope of the data to be used.

You can define the scope of database objects that the AI should consider for generating SQL queries and providing assistance. This scope enables the AI to focus on relevant parts of the database, thereby improving accuracy and efficiency, particularly in large schemas. This is particularly handy when you have two similar schemas or databases, and only one of them is intended for the work.

Memory

Ollama will not impress you with its performance until you have 32GB of memory. My original attempts with 16GB were slow and disappointing. Remember that you are running Ollama and DBeaver at a minimum and maybe a local instance, all of which use memory. I suggest you not attempt using Ollama until you have at least 32GB.

Example

We ask Ollama to create a table on a local PostgreSQL instance

We ask Ollama to create a contact list table

Ollama responds with the DDL for the requested table

Now we request some test data

The table and data from Ollama

Conclusion

DBeaver allows you to use an AI to work without worrying that your prompts will end up on a search engine.

PostgreSQL, Ollama, and the DBeaver AI Assistant

Mon, 28 Jul 2025 15:01:00 +0000

Ollama is an open-source project that simplifies running large language models (LLMs). It runs locally on your machine, and you can choose from multiple LLMs. Keeping all the data on your machine should provide a security bonus.

DBeaver is a universal database tool with an AI Assistant. This assistant provides an extra layer of security by allowing you to lock down what is shared, with the default being metadata only, no data.

Webinar

I recently presented a webinar on using the DBeaver AI Assistant. Several questions were raised about using Ollama, one of the many AI Engines available. But I had not tried it.

The first step is downloading Ollama for your platform (Linux, Windows, or Mac) and installing. Once you have Ollama installed, you will need to pick a LLM. I chose the Gemma3 and installed it using the command line client.

Adding the Gemma3 LLM to Ollama

DBeaver Configuration

Ollama listens on port 11434. I was able to load the Gemma3 model using the 'Load Model' button.

Sample Ollama Settings for Dbeaver EE

Example 1

I started with a simple address book type schema and asked about one last name - @ai who in the address_book has a name last name of Jones?

Finding the Jones

Example 2

The nest test was more of a conversation.

Using the AI Assistant in DBeaver to converse with Ollama

And the results:

Summary

DBeaver works well with Ollama. To me, Ollama is many times slower than the paid AI engines. Now my laptop is not a superpower GPU-laden machine. And maybe the Gemma3 engine is not the optimum for database work. But it does work and is a good place for starting the exploration of your data with an AI.

Writing Queries In English to Create An Address Book On A PostgreSQL Instance

Thu, 24 Jul 2025 13:00:00 +0000

Structured Query Language, or SQL, is my favorite programming language, but its quirky syntax can sometimes be frustrating. Many have professed a need to bypass SQL and query databases in English (another quirky language. Is it possible to have an Artificial Intelligence take a request in English and translate it into SQL on my behalf?

I am using the AI Chat Feature of DBeaver Enterprise to test.

The Prompt

My prompt was Create a table in this postgresql database to hold address book information and then add ten example records from random data to OpenAI.

The Response

Sure, let's create a table named `address_book` with the following fields: `id`, `first_name`, `last_name`, `email`, `phone_number`, and `address`.

CREATE TABLE address_book (

id serial PRIMARY KEY,

first_name varchar(50),

last_name varchar(50),

email varchar(100),

phone_number varchar(15),

address text

);

Now, let's insert ten example records into the `address_book` table. Please note that the data is randomly generated and does not represent real individuals.

INSERT INTO address_book (first_name, last_name, email, phone_number, address) VALUES

('John', 'Doe', '[email protected]', '123-456-7890', '123 Main St, Anytown, USA'),

('Jane', 'Doe', '[email protected]', '234-567-8901', '456 Maple St, Anytown, USA'),

('Jim', 'Smith', '[email protected]', '345-678-9012', '789 Oak St, Anytown, USA'),

('Jill', 'Smith', '[email protected]', '456-789-0123', '012 Pine St, Anytown, USA'),

('Joe', 'Johnson', '[email protected]', '567-890-1234', '345 Birch St, Anytown, USA'),

('Jenny', 'Johnson', '[email protected]', '678-901-2345', '678 Cedar St, Anytown, USA'),

('Jack', 'Williams', '[email protected]', '789-012-3456', '901 Elm St, Anytown, USA'),

('Jill', 'Williams', '[email protected]', '890-123-4567', '234 Spruce St, Anytown, USA'),

('Jerry', 'Brown', '[email protected]', '901-234-5678', '567 Redwood St, Anytown, USA'),

('Jasmine', 'Brown', '[email protected]', '012-345-6789', '890 Willow St, Anytown, USA');

Please replace the example data with your actual data when you use this in a production environment.

The Result

So, I ran a quick SELECT * FROM address_book

1 John Doe [email protected] 123-456-7890 123 Main St, Anytown, USA

2 Jane Doe [email protected] 234-567-8901 456 Maple St, Anytown, USA

3 Jim Smith [email protected] 345-678-9012 789 Oak St, Anytown, USA

4 Jill Smith [email protected] 456-789-0123 012 Pine St, Anytown, USA

5 Joe Johnson [email protected] 567-890-1234 345 Birch St, Anytown, USA

6 Jenny Johnson [email protected] 678-901-2345 678 Cedar St, Anytown, USA

7 Jack Williams [email protected] 789-012-3456 901 Elm St, Anytown, USA

8 Jill Williams [email protected] 890-123-4567 234 Spruce St, Anytown, USA

9 Jerry Brown [email protected] 901-234-5678 567 Redwood St, Anytown, USA

10 Jasmine Brown [email protected] 012-345-6789 890 Willow St, Anytown, USA

Yup, I got exactly what I asked for. The ONLY SQL I used was to query the data after the AI did the 'heavy lifting'.

Analysis

I could quibble that the table structure was not efficient, with the address column defined as TEXT. This would restrict searches by postal code, city, or state. That could easily be fixed by asking the AI in a second round to break out those fields.

Overall, this was a very quick way to create a table and fill it with test data. So, it is possible to have an Artificial Intelligence take a request in English and translate it into SQL on my behalf.

Write PostgreSQL Queries in English?

Wed, 23 Jul 2025 13:53:00 +0000

If you've ever wanted to skip SQL and its quirks and write queries in English, I am happy to report that you can do it with an AI of your choice and DBeaver. I had a webinar on July 24th that shows you how to do just that.

I will show you how to set up DBeaver using a wide variety of commercial AIs to let you work with an AI. We will also go through the security features to ensure your data is safe. By default, only the schema metadata is shared with the AI. You can lock down access to just one or more tables being available to the AI.

Imagine not having to translate a data request from management speak to SQL! Here we simply ask an AI to provide the five most popular films by revenue and get an answer.

I will also show you how an AI can help by explaining and fixing query problems. You can even identify how that 'thrown over a wall' query works.

Have you ever been handed a strange database and have no idea of its content? Then, use an AI to identify its uses.

If you have been looking for a tool to help you with your database, then you need to try DBeaver's AI Assistant.

Watch here

Entity Relationship Maps

Wed, 25 Jun 2025 13:26:00 +0000

Even the most experienced database professionals are known to feel a little anxious when peering into an unfamiliar database. Hopefully, they inspect to see how the data is normalized and how the various tables are combined to answer complex queries. Entity Relationship Maps (ERM) provide a visual overview of how tables are related and can document the structure of the data.

The Community Edition of DBeaver can provide an ERM easily. First, connect to the database. Right-click on the 'Tables' and then select 'View Diagram'.

The ERM is then displayed.

And it gets better! If you are not sure how two tables are joined? Click on the link between the tables, and the names of the columns you will want to use in your JOIN statement are highlighted.

Conclusion

Exploring an unfamiliar database can be daunting. But Entity Relationship Maps provide a way to navigate the territory. And DBeaver is a fantastic tool for working with databases.

Queries on Vacuum

Wed, 30 Apr 2025 15:22:00 +0000

I am (slowly) adding handy PostgreSQL queries to my GitHub, and Vacuum is the newest category. The end goal is to have a compilation of queries for those of us who need to keep an instance healthy.

Over the years, I have collected hundreds of various queries and hate hunting them down in my code snippet library. Finally, they will be in one place and easy to search.

Please contribute if you have similar or better queries (hint, hint!).

Incremental Backups in PostgreSQL 17

Sat, 26 Apr 2025 15:26:00 +0000

The old adage that a DBA or SRE is only as good as their last backup is true. PostgreSQL 17 added the ability to combine multiple incremental backups with a full backup to provide a complete data dictionary to recover a failed system. It is very easy to use.

This is a quick example of using incrementals. I recommend watching this video by Robert Haas for more details and some discussion of backup strategies.

Step 1 - Enable WALL Summerization

The incremental backups require Write Ahead Log Summarization. It is likely not on your instance by default. But it is easy to enable.

demo=# alter system set summerize_wal = 'on';

ALTER SYSTEM

demo=# select pg_reload_conf();

pg_reload_conf

----------------

(1 row)

Step 2 - Full Backup

The examples below are much simpler than you will run into in real life. The scenario is that we make a full backup on Sunday and do incrementals the rest of the week. Wednesday through Saturday are omitted below.

stoker@ThinkPad:~$ mkdir full monday tuesday

stoker@ThinkPad:~$ pg_basebackup -D full --verbose

Password:

pg_basebackup: initiating base backup, waiting for checkpoint to complete

pg_basebackup: checkpoint completed

pg_basebackup: write-ahead log start point: 6/12000028 on timeline 1

pg_basebackup: starting background WAL receiver

pg_basebackup: created temporary replication slot "pg_basebackup_21051"

pg_basebackup: write-ahead log end point: 6/12000158

pg_basebackup: waiting for background process to finish streaming ...

pg_basebackup: syncing data to disk ...

pg_basebackup: renaming backup_manifest.tmp to backup_manifest

pg_basebackup: base backup completed

stoker@ThinkPad:~$

Step 3 - First Incremental (Monday)

On Monday, the first incremental backup is created. The manifest, with all the WAL information on what has changed, needed to be referenced.

stoker@ThinkPad:~$ pg_basebackup -D monday -i ./full/backup_manifest --verbose

Password:

pg_basebackup: initiating base backup, waiting for checkpoint to complete

pg_basebackup: checkpoint completed

pg_basebackup: write-ahead log start point: 6/14000028 on timeline 1

pg_basebackup: starting background WAL receiver

pg_basebackup: created temporary replication slot "pg_basebackup_22703"

pg_basebackup: write-ahead log end point: 6/14000120

pg_basebackup: waiting for background process to finish streaming ...

pg_basebackup: syncing data to disk ...

pg_basebackup: renaming backup_manifest.tmp to backup_manifest

pg_basebackup: base backup completed

Step 4 - Second Incremental (Tuesday)

On Tuesday, another incremental backup is created.

We could have just recorded the changes since the last incremental backup by referring to the backup manifest in the Monday directory. In that case, to completely restore the data, we need to combine the full backup and both the Monday and Tuesday incremental backups. That is a very valid way of operating. If the data does not churn a great deal, this can save time making incrementals.

My personal preference is to base incrementals on full backups. This means only the full and Tuesday backups must be used for complete restoration. This reduction in logical steps may seem trivial, but tempting a Murphy's Law Ticket is something I avoid after too many restorations that are too early in the morning after catastrophic system failures.

stoker@ThinkPad:~$ pg_basebackup -D tuesday -i ./full/backup_manifest --verbose

Password:

pg_basebackup: initiating base backup, waiting for checkpoint to complete

pg_basebackup: checkpoint completed

pg_basebackup: write-ahead log start point: 6/16000028 on timeline 1

pg_basebackup: starting background WAL receiver

pg_basebackup: created temporary replication slot "pg_basebackup_22870"

pg_basebackup: write-ahead log end point: 6/16000120

pg_basebackup: waiting for background process to finish streaming ...

pg_basebackup: syncing data to disk ...

pg_basebackup: renaming backup_manifest.tmp to backup_manifest

pg_basebackup: base backup completed

Step 5 - Combine Full, Monday, and Tuesday

Sometime after the Tuesday incremental backup, something horrible happens to your instance (probably someone with too many instance privileges made a DELETE query without a WHERE clause (AGAIN!?!?)). It is simple to create a new home for the data and combine the Full and Tuesday backup data into that new home.

stoker@ThinkPad:~$ mkdir newdatadir

stoker@ThinkPad:~$ /usr/lib/postgresql/17/bin/pg_combinebackup full tuesday -o newdatadir

stoker@ThinkPad:~$

Step 6 - Restart

Make sure you 'chmod 700' your new datadir and that your postgres account is the owner. Stop the running instance. You're now ready to use the restored data directory. Point pg_ctl at the new datadir, and you are done.

Summary

Incremental backups will make your life easier, and this feature is a big reason for those not on PostgreSQL 17 to upgrade.

A Repository Of Handy Database Administration Queries

Fri, 04 Apr 2025 18:58:00 +0000

I have started a repository of handy SQL Queries for Database Administration, which I have collected over the years. Finding unused or duplicate indexes or queries without indexes and other information can speed up response times. Knowing statistics like cache hit rates can aid in planning upgrades.

I am currently populating this repo as I run across the queries as part of my 'spring cleaning. ' I welcome your participation if you have a handy bit of SQL for an open-source database you'd like to share. The repo is at https://github.com/davestokes/HandySQL, and I will be adding to it over the next few months.

If you have a question that you would like to address, please don't hesitate to let me know. I will try to accommodate you.

Saving The Output From psql

Thu, 27 Mar 2025 20:39:00 +0000

Occasionally, you will need to capture data when working with a database. You can cut-n-paste or use a shell command like script. In typical PostgreSQL fashion, psql has two ways to save your output.

The First Way

The Second Way

The other choice is to use -o <file> or --output=<filename> when you start psql. The differences are that the SQL command is not recorded

Summary

These two little options can save you a lot of grief when you need to save the output from psql. You will find them very handy every so often.

PostgreSQL's COPY and \COPY

Wed, 19 Mar 2025 13:59:00 +0000

PostgreSQL is equivalent to a Swiss Army Knife in the database world. There are things in PostgreSQL that are very simple to use, while in another database, they take many more steps to accomplish. But sometimes, the knife has too many blades, which can cause confusion. This is one of those cases.

COPY and \COPY

I needed a quick dump of a table as part of an ETL (Extract, Transform, and Load - official definition, Exorcism, Trauma, and Lost-sleep - unofficially) to port some data. The COPY command is a quick way to output and write the data to a file. The two copy commands below illustrate adding a delimiter and then writing the output to a table.

demo=# select * from z;
a | b | c
---+----+-----
1 | 10 | 100
2 | 20 | 200
3 | 30 | 300
4 | 40 | 400
(4 rows)
demo=# copy z to stdout (DELIMITER ',');
1,10,100
2,20,200
3,30,300
4,40,400
demo=# copy z to '/tmp/z' (DELIMITER ',');
COPY 4
demo=#

The trick is that you must REMEMBER that the PostgreSQL server is writing the file and will need permission to write in the desired directory. If you try to write your current directory, you will probably see this:

demo=# copy z to 'z' (DELIMITER ',');

ERROR: relative path not allowed for COPY to file

demo=#

ETL

The table I needed to 'fix' as part of the ETL process was edited and was now ready to load. I created a new table.

demo=# create table zz (a1 int, b1 int, c1 int);
CREATE TABLE

The tricky part is that you can not use COPY to load the data. You need to use psql's \COPY! Consider \COPY a wrapper on the client side around the server's COPY command.

demo-# \copy zz from '/tmp/z/' (delimiter ',');
COPY 4
demo-#

I find the COPY - /COPY pairing very handy.

More?

I highly recommend Wager Bianchi's COPY vs. \copy in PostgreSQL and their Security Implications for DBAs for those seeking more information on this subject and its healthy warning about security issues.

Can Artificial Intelligence Created Better Tables Than You?

Sat, 15 Mar 2025 18:08:00 +0000

Artificial Intelligence is one of those conundrums where the ability to have some tasks handled for you contrasts with a mix of ego and pride that it may be better than you at those tasks. I recently wrote a blog using another database about an AI-generated SQL that was quite sophisticated. But what about asking an AI to create a table with a specification like a DBA/SRE/Analyst might receive?

I used Grok and entered the following prompt:

Write the SQL to create a table on a PostgreSQL version 17 server to record data on customers. The data will include a first name, a last name, an address, a birthdate, and a unique UUID primary key. Partition the data by the year of birth. And produce some test data of at least ten records.

I am sure many of us have started large projects given less instruction. Notice: I did not denote the format for the address (US Post Office's format, or UK, or other). Nor did I set a name length. I wanted to see what assumptions were made by the AI.

The Answer

The response came very quickly, much more quickly that I could interactively input the CREATE TABLE statement.

-- First, create the parent table with partitioning

CREATE TABLE customers (
customer_id UUID PRIMARY KEY,
first_name VARCHAR(50) NOT NULL,
last_name VARCHAR(50) NOT NULL,
address TEXT NOT NULL,
birthdate DATE NOT NULL
) PARTITION BY RANGE (EXTRACT(YEAR FROM birthdate));

-- Create partitions for specific year ranges
CREATE TABLE customers_y1960_1980 PARTITION OF customers
FOR VALUES FROM (1960) TO (1980);
CREATE TABLE customers_y1980_2000 PARTITION OF customers
FOR VALUES FROM (1980) TO (2000);
CREATE TABLE customers_y2000_2020 PARTITION OF customers
FOR VALUES FROM (2000) TO (2020);
CREATE TABLE customers_y2020_2040 PARTITION OF customers
FOR VALUES FROM (2020) TO (2040);
-- Insert test data (10 records)
INSERT INTO customers (customer_id, first_name, last_name, address, birthdate) VALUES
-- 1960-1980 partition
(gen_random_uuid(), 'John', 'Smith', '123 Maple St, Boston, MA', '1975-03-15'),
(gen_random_uuid(), 'Mary', 'Johnson', '456 Oak Ave, Chicago, IL', '1968-07-22'),

-- 1980-2000 partition
(gen_random_uuid(), 'Robert', 'Williams', '789 Pine Rd, Seattle, WA', '1985-11-30'),
(gen_random_uuid(), 'Sarah', 'Brown', '321 Elm St, Austin, TX', '1992-04-17'),
(gen_random_uuid(), 'Michael', 'Davis', '654 Birch Ln, Denver, CO', '1998-09-05'),

-- 2000-2020 partition
(gen_random_uuid(), 'Emily', 'Taylor', '987 Cedar Dr, Miami, FL', '2001-12-25'),
(gen_random_uuid(), 'James', 'Wilson', '147 Spruce Way, Portland, OR', '2005-06-10'),
(gen_random_uuid(), 'Lisa', 'Anderson', '258 Willow Ct, Phoenix, AZ', '2010-02-14'),

-- 2020-2040 partition
(gen_random_uuid(), 'Ethan', 'Martinez', '369 Aspen Pl, San Diego, CA', '2021-08-19'),
(gen_random_uuid(), 'Olivia', 'Garcia', '741 Poplar St, New York, NY', '2023-01-03');
-- Verify the data distribution across partitions
SELECT tableoid::regclass, COUNT(*)
FROM customers
GROUP BY tableoid;

Note the last part about selecting the tableoid, which I was not expecting. Maybe a COUNT * on the Primary Key, but not that. Hmmm.

Wow

The response was very quick. A quick scan of the table definition shows some good but probably most efficient choices for the column data types. I specified unique PKs and was unsure if the AI would pop in a UNIQUE constraint or stick with the UUID types ratre conflict strategy. The test data looks good, too, at a cursory glance.

But Does It Work?

The real test is if the supplied SQL works. I copied the CREATE TABLE section and tried.

demo=# CREATE TABLE customers (

customer_id UUID PRIMARY KEY,

first_name VARCHAR(50) NOT NULL,

last_name VARCHAR(50) NOT NULL,

address TEXT NOT NULL,

birthdate DATE NOT NULL

) PARTITION BY RANGE (EXTRACT(YEAR FROM birthdate));

ERROR: unsupported PRIMARY KEY constraint with partition key definition

DETAIL: PRIMARY KEY constraints cannot be used when partition keys include expressions.

demo=#

Dang! Well, everyone runs into syntax and function issues from time to time.

Second Pass

So, maybe it is time to be a little more specific about the birthday data. I modified the prompt to break out the birthday fields into unique columns.

Write the SQL to create a table on a PostgreSQL version 17 server to record data on customers. The data will include a first name, a last name, an address, a birthdate, and a unique UUID primary key. Have separate columns for the birthday month, birthday day, and birthday year. Partition the data by the year of birth. And produce some test data from at least ten records.

Once again, the response came quickly.

-- Create the parent table with partitioning

CREATE TABLE customers (

customer_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),

first_name VARCHAR(50) NOT NULL,

last_name VARCHAR(50) NOT NULL,

address TEXT NOT NULL,

birth_date DATE NOT NULL,

birth_month SMALLINT GENERATED ALWAYS AS (EXTRACT(MONTH FROM birth_date)) STORED,

birth_day SMALLINT GENERATED ALWAYS AS (EXTRACT(DAY FROM birth_date)) STORED,

birth_year SMALLINT GENERATED ALWAYS AS (EXTRACT(YEAR FROM birth_date)) STORED

) PARTITION BY RANGE (birth_year);

-- Create partitions for specific year ranges

CREATE TABLE customers_y1970s PARTITION OF customers

FOR VALUES FROM (1970) TO (1980);

CREATE TABLE customers_y1980s PARTITION OF customers

FOR VALUES FROM (1980) TO (1990);

CREATE TABLE customers_y1990s PARTITION OF customers

FOR VALUES FROM (1990) TO (2000);

CREATE TABLE customers_y2000s PARTITION OF customers

FOR VALUES FROM (2000) TO (2010);

CREATE TABLE customers_y2010s PARTITION OF customers

FOR VALUES FROM (2010) TO (2020);

-- Insert test data

INSERT INTO customers (first_name, last_name, address, birth_date) VALUES

('John', 'Smith', '123 Main St, Boston MA', '1985-03-15'),

('Sarah', 'Johnson', '456 Oak Ave, Chicago IL', '1992-07-22'),

('Michael', 'Brown', '789 Pine Rd, Seattle WA', '1978-11-30'),

('Emily', 'Davis', '321 Elm St, Austin TX', '2001-02-14'),

('David', 'Wilson', '654 Cedar Ln, Miami FL', '1988-09-05'),

('Lisa', 'Anderson', '987 Birch Dr, Denver CO', '1995-12-25'),

('James', 'Taylor', '147 Maple St, Portland OR', '1975-06-18'),

('Rachel', 'Martinez', '258 Willow Way, Phoenix AZ', '2004-04-03'),

('Thomas', 'Lee', '369 Spruce Ct, Atlanta GA', '1982-08-27'),

('Kelly', 'White', '741 Ash Pl, San Diego CA', '1998-10-11'),

('Robert', 'Garcia', '852 Poplar Ave, Houston TX', '1972-01-09');

-- Verify the data

SELECT * FROM customers ORDER BY birth_year, birth_month, birth_day;

The choice of generated columns for breaking out the birthdate information looked like a good choice. I was surprised to see the last query changed from counting the rows to selecting all the data. What triggered that?

Now, does that new CREATE TABLE work?

Close, Lacking Cigar

I had high hopes that the second revision would work. The date was split up into individual columns.

demo=# CREATE TABLE customers (

customer_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),

first_name VARCHAR(50) NOT NULL,

last_name VARCHAR(50) NOT NULL,

address TEXT NOT NULL,

birth_date DATE NOT NULL,

birth_month SMALLINT GENERATED ALWAYS AS (EXTRACT(MONTH FROM birth_date)) STORED,

birth_day SMALLINT GENERATED ALWAYS AS (EXTRACT(DAY FROM birth_date)) STORED,

birth_year SMALLINT GENERATED ALWAYS AS (EXTRACT(YEAR FROM birth_date)) STORED

) PARTITION BY RANGE (birth_year);

ERROR: cannot use generated column in partition key

LINE 10: ) PARTITION BY RANGE (birth_year);

DETAIL: Column "birth_year" is a generated column.

demo=#

Dang! I did not know that using a GENERATED COLUMN as an argument for partition would be an issue. (Time for me and the AI to RTFM, right?)

One More Into The Breech

So this time, I added 'do not use generated columns'.

Write the SQL to create a table on a PostgreSQL version 17 server to record data on customers. The data will include a first name, a last name, an address, a birthdate, and a unique UUID primary key. Have separate columns, do not use generated columns, for the birthday month, birthday day, and birthday year. Partition the data by the year of birth. And produce some test data from at least ten records.

-- Create the parent table with partitioning

CREATE TABLE customers (

customer_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),

first_name VARCHAR(50) NOT NULL,

last_name VARCHAR(50) NOT NULL,

address TEXT NOT NULL,

birth_date DATE NOT NULL,

birth_month SMALLINT NOT NULL CHECK (birth_month >= 1 AND birth_month <= 12),

birth_day SMALLINT NOT NULL CHECK (birth_day >= 1 AND birth_day <= 31),

birth_year SMALLINT NOT NULL

) PARTITION BY RANGE (birth_year);

-- Create partitions for specific year ranges

CREATE TABLE customers_y1970s PARTITION OF customers

FOR VALUES FROM (1970) TO (1980);

CREATE TABLE customers_y1980s PARTITION OF customers

FOR VALUES FROM (1980) TO (1990);

CREATE TABLE customers_y1990s PARTITION OF customers

FOR VALUES FROM (1990) TO (2000);

CREATE TABLE customers_y2000s PARTITION OF customers

FOR VALUES FROM (2000) TO (2010);

CREATE TABLE customers_y2010s PARTITION OF customers

FOR VALUES FROM (2010) TO (2020);

-- Insert test data

INSERT INTO customers (first_name, last_name, address, birth_date, birth_month, birth_day, birth_year) VALUES

('John', 'Smith', '123 Main St, Boston MA', '1985-03-15', 3, 15, 1985),

('Sarah', 'Johnson', '456 Oak Ave, Chicago IL', '1992-07-22', 7, 22, 1992),

('Michael', 'Brown', '789 Pine Rd, Seattle WA', '1978-11-30', 11, 30, 1978),

('Emily', 'Davis', '321 Elm St, Austin TX', '2001-02-14', 2, 14, 2001),

('David', 'Wilson', '654 Cedar Ln, Miami FL', '1988-09-05', 9, 5, 1988),

('Lisa', 'Anderson', '987 Birch Dr, Denver CO', '1995-12-25', 12, 25, 1995),

('James', 'Taylor', '147 Maple St, Portland OR', '1975-06-18', 6, 18, 1975),

('Rachel', 'Martinez', '258 Willow Way, Phoenix AZ', '2004-04-03', 4, 3, 2004),

('Thomas', 'Lee', '369 Spruce Ct, Atlanta GA', '1982-08-27', 8, 27, 1982),

('Kelly', 'White', '741 Ash Pl, San Diego CA', '1998-10-11', 10, 11, 1998),

('Robert', 'Garcia', '852 Poplar Ave, Houston TX', '1972-01-09', 1, 9, 1972);

-- Verify the data

SELECT * FROM customers ORDER BY birth_year, birth_month, birth_day;

No generated columns and some constrict checks added gratis. Now, the final query has been augmented with an ORDER BY.

But, once again, does IT work?

You Make Take A Prize From The Losest Shelf

demo=# CREATE TABLE customers (

customer_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),

first_name VARCHAR(50) NOT NULL,

last_name VARCHAR(50) NOT NULL,

address TEXT NOT NULL,

birth_date DATE NOT NULL,

birth_month SMALLINT NOT NULL CHECK (birth_month >= 1 AND birth_month <= 12),

birth_day SMALLINT NOT NULL CHECK (birth_day >= 1 AND birth_day <= 31),

birth_year SMALLINT NOT NULL

) PARTITION BY RANGE (birth_year);

ERROR: unique constraint on partitioned table must include all partitioning columns

DETAIL: PRIMARY KEY constraint on table "customers" lacks column "birth_year" which is part of the partition key.

demo=#

Okay, I need to read up on the partitioning by range manual page.

Conclusions

The DDL definition looks good at first glance. And it is almost right, which may catch the attention of many of us who write SQL for a living. Replace the AI with a Junior DBA in your mind, and you would consider the results pretty good. Now, to get the AI to RTFM about partitioning and generated columns, the results will be stupifying. I am sure there are some clever folks working to include the source and manuals for PostgreSQL into a model so the issues I had above will disappear.

IDEs have become essential tools for many developers, with features like syntax completion. Could the future include a similar collection of features where we supply the DDL and/or DML spec and the AI returns what was requested?

One of my first programming classes featured a section on unexpressed latent desires in specifications. Earlier, I mentioned not specifying a format for addresses. I have been on a project where the multinationalization of addresses became a headache, and maybe the AI could anticipate that issue before the developers notice it with the proper design to accommodate the situation.

I am impressed how well Grok did with my requirements. I am sure it will only get better as time goes on. Do not automatically reject the idea that an AI can produce good SQL. Just like we trust compilers to take our thoughts and turn them into machine code, AI will soon turn our database ideas and create the required DML and DDL.

Update

After finishing this blog's first pass, I was drinking a coffee when the brain asked, 'What if I drop the partitioning requirement to the original query and try again?' The results:

-- Create the customers table

CREATE TABLE customers (

customer_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),

first_name VARCHAR(50) NOT NULL,

last_name VARCHAR(50) NOT NULL,

address TEXT NOT NULL,

birth_date DATE NOT NULL

);

-- Insert test data

INSERT INTO customers (first_name, last_name, address, birth_date) VALUES

('John', 'Smith', '123 Main St, Boston, MA 02108', '1985-06-15'),

('Sarah', 'Johnson', '456 Oak Ave, Chicago, IL 60601', '1990-03-22'),

('Michael', 'Brown', '789 Pine Rd, Austin, TX 73301', '1978-11-30'),

('Emily', 'Davis', '321 Elm St, Seattle, WA 98101', '1995-09-12'),

('David', 'Wilson', '654 Cedar Ln, Miami, FL 33101', '1982-04-18'),

('Lisa', 'Anderson', '987 Birch Dr, Denver, CO 80201', '1988-12-05'),

('James', 'Taylor', '147 Maple Way, Portland, OR 97201', '1975-07-25'),

('Rachel', 'Martinez', '258 Spruce Ct, Phoenix, AZ 85001', '1993-01-09'),

('Thomas', 'Lee', '369 Willow Pl, Atlanta, GA 30301', '1980-10-14'),

('Kelly', 'White', '741 Aspen St, San Diego, CA 92101', '1987-08-03');

I ran a quick query:

demo=# select first_name, last_name, birth_date

from customers order by last_name, first_name;

first_name | last_name | birth_date

------------+-----------+------------

Lisa | Anderson | 1988-12-05

Michael | Brown | 1978-11-30

Emily | Davis | 1995-09-12

Sarah | Johnson | 1990-03-22

Thomas | Lee | 1980-10-14

Rachel | Martinez | 1993-01-09

John | Smith | 1985-06-15

James | Taylor | 1975-07-25

Kelly | White | 1987-08-03

David | Wilson | 1982-04-18

(10 rows)

Grok added a note at the end of the code that I had not concentrated on before this iteration,

This SQL code:

Creates a table named "customers" with:

customer_id: A UUID primary key that auto-generates using PostgreSQL 17's gen_random_uuid() function
first_name: A required variable-length string (up to 50 characters)
last_name: A required variable-length string (up to 50 characters)
address: A required text field for full addresses
birth_date: A required date field

Inserts 10 test records with realistic sample data:

Varied names
Addresses in different US cities with plausible ZIP codes
Birth dates spanning different years from 1975 to 1995

The UUID values will be automatically generated when you run the INSERT statement. Each record will get a unique UUID in the format like '550e8400-e29b-41d4-a716-446655440000'.

You can run this SQL directly in a PostgreSQL 17 database to create the table and populate it with the test data.

So, is this a good thing or a bad thing? Grok was very close to making my socks fly off. I submit that the partitioning issue is a minor issue. Maybe some extra specification on my part could guide it past the rocky shoals of the partitioning definition waters.

Use PASSING with JSON_TABLE() To Make Calculations

Tue, 25 Feb 2025 15:29:00 +0000

I ran across this trick while preparing a talk for the Southern California Linux Expo (SCaLE) on using JSON_TABLE(). This is a 'reserve' talk, which means it will presented IF another speaker drops out. But I thought this was interesting enough to share here. I will make sure the slides are posted online when finished.

Passing Interest

JSON_TABLE() temporarily converts JSON data into a relational table so that the temporary table can be processed by Structured Query Langauge (SQL) Commands. A feature in the PostgreSQL implementation is the passing clause. In the example below, you will see PASSING 1500 AS price_limit in red. This declares price_limit as a variable and assigns a value of 1500 to it.

Later, in the body of the JSON_TABLE() call, you will see in green is_too_expensive BOOLEAN EXISTS PATH '$.price ? (@ > $price_limit)'. Here a variable is created named is_too_expensive of type BOOLEAN, which means we get a true or false answer from the rest of the declaration. If the price column has a value, it is compared with the price_limit value to see if that value is greater or not than the limit.

demo=# SELECT *
FROM JSON_TABLE(
'{"guitars":[{"item": "Fender Telecaster", "price": 1000},
{"item": "Gibson Les Paul", "price": 2000},
{"item": "PRS semi-hollowbody", "price": 1500}]}',
'$.guitars[*]' PASSING 1500 AS price_limit
COLUMNS (
row_num for ordinality,
Guitar TEXT PATH '$.item',
Price INT PATH '$.price',
is_too_expensive BOOLEAN EXISTS PATH '$.price ? (@ > $price_limit)'
)
) AS my_example;
row_num | guitar | price | is_too_expensive
---------+---------------------+-------+------------------
1 | Fender Telecaster | 1000 | f
2 | Gibson Les Paul | 2000 | t
3 | PRS semi-hollowbody | 1500 | f
(3 rows)
demo=#

How Does This Help You?

JSON data is tough on relational databases as it can not be processed with the regular operators found in SQL without special calls. Those calls can slow down a query tremendously. We could have pulled the data from JSON and then used another query to make our quantitative determination. But that is another query on top of the JSON_TABLE() call, and two queries are going to be slower than one. Or we could have used some sort of case statement. But this is such a simple calculation that it seems tragic to separate it from the rest of the data.

This is a quick and 'not dirty' to handle the data.

And if you are at SCaLE, please day 'hello!'

How PostgreSQL's Aggregate FILTER Will Spoil You

Sat, 22 Feb 2025 15:41:00 +0000

Using PostgreSQL will definitely spoil you. I recently had to use another database and found, to my chagrin, that aggregate filters were not included in their implementation of Structured Query Language. Specifically, there was no FILTER clause available.

FILTER?

To illustrate my dependency, let me start with some simple data.

demo=# create table z (a int, b int, c int);
CREATE TABLE
demo=# insert into z values (1,10,100),(2,20,200),(3,30,300),(4,40,400);
INSERT 0 4
demo=# select a,b,c from z;
a | b | c
---+----+-----
1 | 10 | 100
2 | 20 | 200
3 | 30 | 300
4 | 40 | 400
(4 rows)
demo=#

If I wanted the overall number of rows, I could use a simple COUNT(*). If I wanted an overall number of rows where one of the columns was of a certain value, I could add the logic for getting those rows to the WHERE clause.

What if I wanted both? Do I need two queries to do the work separately? Maybe a materialized view or a CTE (Common Table Expression)? Will I need a Window Function?

FILTER!

PostgreSQL provides the ability to do what is needed with one SELECT statement.

demo=# select count(*) as all_rows,

demo-# count(*) FILTER (where b > 11) as b_gt_11

demo-# from z;

all_rows | b_gt_11

----------+---------

4 | 3

(1 row)

demo=#

The above SELECT statement asks for two columns of data. The first asks for a count of all rows.

The second requests a count of all rows where the b column's value exceeds 11. The FILTER defines the rows of interest.

Long-time SQL writers will immediately spot how handy the ability to filter rows this way is.

Sadly, the database I had to use did not offer the ability to do this. What should have been a simple query became a detour into the bumpy backroads of finding another way to get the answer.

Constraint Checks And Dirty Data

Wed, 19 Feb 2025 17:37:00 +0000

It is much easier to keep bad data out of your database than it is to have to correct it later. Those costs are measured in time, money, and sanity. One of the easier ways of protecting your data is with a constraint check.

I was recently approached by someone who urgently needed a constraint check and could afford to 'fix' the bad data after the constraint was in place. They had to quickly keep new 'bad' data out and could afford to correct the old 'bad' data at leisure.

The Overly Simplified Example

To illustrate how to do just that, here is an example of a table and some simple data.

demo=# create table x1 (a int);
CREATE TABLE
demo=# insert into x1 (a) values (10),(20),(30),(-5),(1);
INSERT 0 5

demo=#

Sometime later, it is discovered that any value of a needs to be greater than five. If we add the constraint, the system will balk and refuse the request.

demo=# alter table x1 add constraint x_id_gt_5 check (a > 5);
ERROR: check constraint "x_id_gt_5" of relation "x1" is violated by some row

The trick is to tell the ALTER TABLE command to not validate the constraint check. The constraint is added, but the system does not ensure the data conforms to that check. This is done by adding NOT VALID to the ALTER TABLE command.

demo=# alter table x1 add constraint a_gt_5 check (a > 5) not valid;
ALTER TABLE

The data is in the same state as before the constraint was added. The old 'bad' data is still there and can be corrected or deleted later.

demo=# select a from x1;

----

-5

(5 rows)

demo=#

Any new attempts to add new 'bad' data will fail.

demo=# insert into x1 (a) values (3);

ERROR: new row for relation "x1" violates check constraint "a_gt_5"

DETAIL: Failing row contains (3).

demo=#

Now, it is time to fix the rows that do not conform to the check. When that is done, the check can be validated.

demo=# alter table x1 validate constraint a_gt_5;

ALTER TABLE

demo=#

Constraint Hints

1. Name your constraints. The name 'a_gt_5' describes the constraint and explains why the data was rejected. If you do not name the constraint, the system will assign a name that will list the table name, the column name, the word 'check,' and no clue what went wrong.

Look at the following definitions. Seeing a_gt_t error informs you what is wrong, But x1_a_check is nebulous, at best. Remember that you may be the poor person trying to figure out why the inserts are failing at three in the morning.

demo=# \d x1

Table "public.x1"

Column | Type | Collation | Nullable | Default

--------+---------+-----------+----------+---------

a | integer | | |

Check constraints:

"a_gt_5" CHECK (a > 5)

"x1_a_check" CHECK (a > 6)

demo=#

2. Double-check the constraints. In the above, having a check on a being greater than five and another greater than six makes no sense. This type of problem happens as the tables age and multiple people make changes without making sure there no conflict or similar action already established.

PostgreSQL MERGE To Reconcile Cash Register Sales Data Part 2

Mon, 17 Feb 2025 14:01:00 +0000

Part one of this series covered a simple use of MERGE and can be found here. In Part Two, we get more complicated logically. Previously, we only had a WHEN NOT MATCHED logical leg in the code. Now, we add a WHEN MATCHED logic.

WHEN MATCHED and WHEN NOT MATCHED

The query now has two logical legs - when there are matching rows and when there are no matching rows. I have color-coded the query to illustrate the separate paths.

demo=# merge into ledger l
using register r
on l.id = r.id
WHEN NOT MATCHED THEN
insert (id, tx_id, total_register, misc)
values (r.id, r.tx_id, r.amount, 0)
WHEN MATCHED THEN
update set misc = misc + 1
RETURNING merge_action(), l.*;
~
merge_action | id | tx_id | total_register | misc
--------------+----+-------+----------------+------
UPDATE | 1 | 101 | $19.99 | 1
(1 row)
MERGE 1
demo-#

The ledger table already has a match in the above example, so we increment the misc column when the query runs. If we rerun the query, the same column gets incremented again.

demo=# merge into ledger l

using register r

on l.id = r.id

WHEN NOT MATCHED THEN

insert (id, tx_id, total_register, misc)

values (r.id, r.tx_id, r.amount, 0)

WHEN MATCHED THEN

update set misc = misc + 1

RETURNING merge_action(), l.*;

merge_action | id | tx_id | total_register | misc

--------------+----+-------+----------------+------

UPDATE | 1 | 101 | $19.99 | 2

(1 row)

MERGE 1

demo-#

Delete

Another option with Merge is to DELETE the row. The NOT MATCHED logic does not change but there is now a WHEN MATCHED logical leg.

demo=# merge into ledger l

using register r

on l.id = r.id

WHEN NOT MATCHED THEN

insert (id, tx_id, total_register, misc)

values (r.id, r.tx_id, r.amount, 0)

WHEN MATCHED THEN

DELETE

RETURNING merge_action(), l.*;

merge_action | id | tx_id | total_register | misc

--------------+----+-------+----------------+------

DELETE | 1 | 101 | $19.99 | 2

(1 row)

MERGE 1

demo=# select * from ledger;

id | tx_id | total_register | misc

----+-------+----------------+------

(0 rows)

demo=#

Doing Nothing

There is also an option to DO NOTHING. Yes, you can get paid to write code that does nothing.

Remember that the ledger table is empty after the last example. We have to run the code one to insert a row into this table. Then, we run it again to test the WHEN MATCHED logic leg with the DO NOTHING option.

demo=# merge into ledger l

using register r

on l.id = r.id

WHEN NOT MATCHED THEN

insert (id, tx_id, total_register, misc)

values (r.id, r.tx_id, r.amount, 0)

WHEN MATCHED THEN

DO NOTHING

RETURNING merge_action(), l.*;

merge_action | id | tx_id | total_register | misc

--------------+----+-------+----------------+------

INSERT | 1 | 101 | $19.99 | 0

(1 row)

MERGE 1

demo=# merge into ledger l

using register r

on l.id = r.id

WHEN NOT MATCHED THEN

insert (id, tx_id, total_register, misc)

values (r.id, r.tx_id, r.amount, 0)

WHEN MATCHED THEN

DO NOTHING

RETURNING merge_action(), l.*;

merge_action | id | tx_id | total_register | misc

--------------+----+-------+----------------+------

(0 rows)

MERGE 0

demo=#

Note that the merge_action() function does not say 'Nothing'.

Checking the ledger table shows us one row.

demo=# select * from ledger;

id | tx_id | total_register | misc

----+-------+----------------+------

1 | 101 | $19.99 | 0

(1 row)

demo=#

More Complicated Logic

We can add complications to the logic to make the query more useful. We will add a status column to the ledger table to clarify the illustration.

demo=# alter table ledger add column status text;

ALTER TABLE

demo=# truncate ledger;

TRUNCATE TABLE

demo=#

The new logic for the query is just a little more complicated.

No Match

Insert data

There is a match, but it is the first time we match

Update the status field

There is a match, and it is the second time we match

Delete the row

The first run puts 'inserted' into the status column.

demo=# merge into ledger l

using register r

on l.id = r.id

WHEN NOT MATCHED THEN

insert (id, tx_id, total_register, misc, status)

values (r.id, r.tx_id, r.amount, 0, 'inserted')

WHEN MATCHED and l.status = 'inserted' THEN

update set misc = misc + 1, status = 'already seen'

WHEN MATCHED and l.status = 'already seen' THEN

DELETE

RETURNING merge_action(), l.*;

--------------+----+-------+----------------+------+----------

INSERT | 1 | 101 | $19.99 | 0 | inserted

(1 row)

MERGE 1

demo-#

The second pass updates the status column to 'already seen'.

demo=# merge into ledger l

using register r

on l.id = r.id

WHEN NOT MATCHED THEN

insert (id, tx_id, total_register, misc, status)

values (r.id, r.tx_id, r.amount, 0, 'inserted')

WHEN MATCHED and l.status = 'inserted' THEN

update set misc = misc + 1, status = 'already seen'

WHEN MATCHED and l.status = 'already seen' THEN

DELETE

RETURNING merge_action(), l.*;

--------------+----+-------+----------------+------+--------------

UPDATE | 1 | 101 | $19.99 | 1 | already seen

(1 row)

MERGE 1

The third pass of the query where status equals 'already seen' will DELETE the row.

demo=# merge into ledger l

using register r

on l.id = r.id

WHEN NOT MATCHED THEN

insert (id, tx_id, total_register, misc, status)

values (r.id, r.tx_id, r.amount, 0, 'inserted')

WHEN MATCHED and l.status = 'inserted' THEN

update set misc = misc + 1, status = 'already seen'

WHEN MATCHED and l.status = 'already seen' THEN

DELETE

RETURNING merge_action(), l.*;

--------------+----+-------+----------------+------+--------------

DELETE | 1 | 101 | $19.99 | 1 | already seen

(1 row)

MERGE 1

demo=# select * from ledger;

id | tx_id | total_register | misc | status

----+-------+----------------+------+--------

(0 rows)

demo=#

Summary

Merge is very handy for tasks like cash register sales reconciliation. You can make the WHEn MATCHED or WHEN NOT MATCHED logic as complicated as needed. Please peruse the manual page as it provides more details on its use.

In 'real life,' you probably want a primary key that is complex enough to be unique, like a store number, register number, and transaction number to ensure uniqueness. Make sure you use techniques like constraint checks on the data, as it is easier and cheaper to out than try to go back later to correct it.

If you have an interesting use for Merger and can share it, please let me know.

PostgreSQL MERGE To Reconcile Cash Register Sales Data Part 1

Tue, 11 Feb 2025 16:14:00 +0000

I hoped to present the following at the PostgreSQL Conference 2025 in March. But I can not make it to Orlando for this wonderful event (Please attend if you can! It is a fantastic event). Cash Register information is vital to retail businesses and it is often processed one line item at a time (pronounced 'S-L-O-W'). MERGE was added a few years ago to PostgreSQL's arsenal. It is powerful and easy to use.

The Cash Register

The next time you are handed a receipt from a cash register, take the time to examine all the information on it. Time, date, location, total, line items, and possibly other information essential to the vendor's operations are on that slip of paper. There may be information not on the sales receipt collected for processing. Now imagine you are a major fast-food restaurant with tens of thousands of cash registers or more. All that data needs to be collected, transmitted, and processed. It is easy to see how efficiency is important.

MERGE

The manual page describes the function of MERGE distinctly as MERGE performs actions that modify rows in the target table identified as target_table_name, using the data_source. MERGE provides a single SQL statement that can conditionally INSERT, UPDATE or DELETE rows, a task that would otherwise require multiple procedural language statements. In the examples below the data source is the data from the cash register and the target table is the sales table.

Our simple example uses the following table for cash register data. We will assume that each cash register in our imaginary fast-food chain has a unique identification number.

demo=# create table register (id int, tx_id int, amount money);
CREATE TABLE
demo=# insert into register values (1,101,19.99);
INSERT 0 1
demo=#

A table to record all the transactions is created. Not that this ledger table is empty and our first sale is ready to be processed.

demo=# create table ledger (id int, tx_id int, total_register money, misc int);

CREATE TABLE

demo=# select * from register;

id | tx_id | amount

----+-------+--------

1 | 101 | $19.99

(1 row)

demo=# select * from ledger;

id | tx_id | total_register | misc

----+-------+----------------+------

(0 rows)

demo=#

First Merge

The first time we merge, the code is simple. We compare the entries in the register table to see if there are matches in the ledger table. There are no entries in the leger table. A clause in the code moves the data from the register table into the ledger table if there are NO MATCHES. We end with the RETURNING line to report what has been performed (the INSERT) and the new values in the ledger table.

demo=# merge into ledger l

using register r

on l.id = r.id

WHEN NOT MATCHED THEN

insert (id, tx_id, total_register, misc)

values (r.id, r.tx_id, r.amount, 0)

RETURNING merge_action(), l.*;

merge_action | id | tx_id | total_register | misc

--------------+----+-------+----------------+------

INSERT | 1 | 101 | $19.99 | 0

(1 row)

MERGE 1

demo=#

Second Merge

If we run the same SQL code again, the results look a little different even if the data in the ledger tables does not change. This time there IS a match between the two tables. Later you will see how to right WHEN MATCH clauses.

Sometimes the same data is applied twice which is a disaster in many cases. But in this case, there was no action taken.

What is needed is to compare the transaction IDs along with the register IDs to make sure we do not record twice the same line of data. heck, we probably in real life want a unique key made with the store number and the register id.

Next Time

I am trying to keep these blogs short, if not sweet. And this one went on a bit longer than I thought. Next time we will look at more MERGE() functionality and explore the logic of making matches to optimize the performance of the query.