F.52. pg_query_id#

F.52. pg_query_id

F.52. pg_query_id #

Extension to manipulate aspects of query id computation.

F.52.1. About pg_query_id #

Version: 1.0.0

F.52.2. Getting started #

Caution

Extension must be last in shared_preload_libraries or at least last extension that uses query id. There is not check for that, so be careful.

Install extension by adding it to shared_preload_libraries and set GUC to ignore arrays of constants. And also add pg_stat_statements for further checking.

shared_preload_libraries = 'pg_query_id'
pg_query_id.ignore_array_of_constants = on

Start DB and run these 2 queries:

SELECT * FROM generate_series(1, 100) x WHERE x IN (101, 102, 103);
SELECT * FROM generate_series(1, 100) x WHERE x IN (101, 102, 103, 104, 105);

pg_stat_statements must contain single query with only 3 constants (other queries omitted):

SELECT query FROM pg_stat_statements;

                              query                              
-----------------------------------------------------------------
 SELECT * FROM generate_series($1, $2) x WHERE x IN ($3, $4, $5)

F.52.3. Functionality #

Note

For checking query id we use pg_stat_statements in examples

To change configuration below, change configuration files and then send SIGHUP.

F.52.3.1. Enabling/disabling #

GUC Default
pg_query_id.enabled on

This setting controls, whether extension is up and running. Using this you can disable extension at runtime, but note that computed query ids will be different.

F.52.3.2. Temporary table names ignoring #

GUC Default
pg_query_id.ignore_temp_tables off

If this setting is on, then OIDs of temporary tables are not used during query id computation. This leads to same query ids for queries, where only temporary tables differs, but rest of the query is the same.

Example:

CREATE TEMP TABLE temp_tbl1(id int, str text);
CREATE TEMP TABLE temp_tbl2(id int, str text);

SELECT * FROM temp_tbl1 WHERE id = 1;
SELECT * FROM temp_tbl2 WHERE id = 1;

SELECT query FROM pg_stat_statements;

                     query                     
-----------------------------------------------
 CREATE TEMP TABLE temp_tbl1(id int, str text)
 SELECT * FROM temp_tbl1 WHERE id = $1

As you can see only first executed query was recorded and second executed query (with only different table name) has same query id so go into same tuple.

F.52.3.3. Ignore arrays of constants #

GUC Default
pg_query_id.ignore_constant_arrays off

If this setting is on, then arrays of only constants (in different queries with different length of array) will not contribute to query id computation. This includes:

  • ARRAY[]

  • VALUES () (if every row has only constants)

  • IN ()

If such construction contains CONSTANTs or IMPLICIT CASTs OF CONSTANT then it will be ignored. I.e. if you have SELECT 1 inside array it will NOT be ignored and computed by default.

Example:

SELECT * FROM generate_series(1, 10) x WHERE x IN (11, 12, 13);
SELECT * FROM generate_series(1, 10) x WHERE x IN (11, 12, 13, 14, 15);

SELECT * FROM generate_series(1, 10) x WHERE x = ANY(VALUES (11), (12), (13));
SELECT * FROM generate_series(1, 10) x WHERE x = ANY(VALUES (11), (12), (13), (14), (15));

SELECT * FROM generate_series(1, 10) x WHERE x = ALL(ARRAY[11, 12, 13]);
SELECT * FROM generate_series(1, 10) x WHERE x = ALL(ARRAY[11, 12, 13, 14, 15]);

SELECT query FROM pg_stat_statements;

                                           query                                            
--------------------------------------------------------------------------------------------
 SELECT * FROM generate_series($1, $2) x WHERE x = ALL(ARRAY[$3, $4, $5])
 SELECT * FROM generate_series($1, $2) x WHERE x = ANY(VALUES ($3), ($4), ($5))
 SELECT * FROM generate_series($1, $2) x WHERE x IN ($3, $4, $5)

But not only constants, but implicit cast also work:

CREATE TABLE tbl(id int, str char(20));
SELECT * FROM tbl WHERE str IN ('a', 'b', 'c');
SELECT * FROM tbl WHERE str IN ('a', 'b', 'c', 'd', 'e');

SELECT query FROM pg_stat_statements;

                    query                    
---------------------------------------------
 SELECT * FROM tbl WHERE str IN ($1, $2, $3)
 CREATE TABLE tbl(id int, str char(20))

But this will NOT work:

SELECT * FROM generate_series(1, 10) x WHERE x IN (11, 12, (SELECT 13));
SELECT * FROM generate_series(1, 10) x WHERE x IN (11, 12, 13, (SELECT 14));

SELECT * FROM generate_series(1, 10) x WHERE x = ANY(VALUES (11), (12), ((SELECT 13)));
SELECT * FROM generate_series(1, 10) x WHERE x = ANY(VALUES (11), (12), (13), ((SELECT 14)));

SELECT * FROM generate_series(1, 10) x WHERE x = ALL(ARRAY[11, 12, (SELECT 13)]);
SELECT * FROM generate_series(1, 10) x WHERE x = ALL(ARRAY[11, 12, 13, (SELECT 14)]);

SELECT query FROM pg_stat_statements;

                                             query                                             
-----------------------------------------------------------------------------------------------
 SELECT * FROM generate_series($1, $2) x WHERE x IN ($3, $4, (SELECT $5))
 SELECT * FROM generate_series($1, $2) x WHERE x IN ($3, $4, $5, (SELECT $6))
 SELECT * FROM generate_series($1, $2) x WHERE x = ALL(ARRAY[$3, $4, (SELECT $5)])
 SELECT * FROM generate_series($1, $2) x WHERE x = ALL(ARRAY[$3, $4, $5, (SELECT $6)])
 SELECT * FROM generate_series($1, $2) x WHERE x = ANY(VALUES ($3), ($4), ((SELECT $5)))
 SELECT * FROM generate_series($1, $2) x WHERE x = ANY(VALUES ($3), ($4), ($5), ((SELECT $6)))

F.52.3.4. Constant query id for utility statements #

GUC Default
pg_query_id.utility_constant_query_id off

If this setting is on, then query id for some types of utility statement types assigned with predefined constant. Query id assignment table:

Utility statement Query Id
CREATE TABLE 1
CREATE TEMP TABLE 2
CREATE TABLE ... AS 3
CREATE TEMP TABLE ... AS 4
CREATE INDEX 5
CREATE INDEX (temp table) 6
VACUUM 7
TRUNCATE 8
DROP TABLE 9
DROP INDEX 10
COPY FROM ... (temp table) 11
COPY FROM ... 12
COPY TO ... (temp table) 13
COPY TO ... 14

This functionality may be useful for coarse-grained statistics collection when we want to measure overhead of multiple DMLs.

Example:

CREATE TABLE tbl1(id int);
CREATE TABLE tbl2(id int, str text);

CREATE TEMP TABLE temp_tbl1(id int);
CREATE TEMP TABLE temp_tbl2(id int, str text);

CREATE TABLE tbl3 AS SELECT x FROM generate_series(1, 10) x;
CREATE TABLE tbl4 AS SELECT x, x::text y FROM generate_series(1, 10) x;

CREATE TEMP TABLE temp_tbl3 AS SELECT x FROM generate_series(1, 10) x;
CREATE TEMP TABLE temp_tbl4 AS SELECT x, x::text y FROM generate_series(1, 10) x;

CREATE INDEX tbl1_id_idx ON tbl1(id);
CREATE INDEX temp_tbl1_id_idx ON temp_tbl1(id);

VACUUM tbl1;
VACUUM temp_tbl1;

COPY tbl1 TO STDOUT;
COPY tbl2 TO STDOUT;
COPY tbl3 TO STDOUT;
COPY tbl4 TO STDOUT;
COPY temp_tbl1 TO STDOUT;
COPY temp_tbl2 TO STDOUT;
COPY temp_tbl3 TO STDOUT;
COPY temp_tbl4 TO STDOUT;

COPY tbl1 FROM STDIN;
COPY tbl2 FROM STDIN;
COPY tbl3 FROM STDIN;
COPY tbl4 FROM STDIN;
COPY temp_tbl1 FROM STDIN;
COPY temp_tbl2 FROM STDIN;
COPY temp_tbl3 FROM STDIN;
COPY temp_tbl4 FROM STDIN;

TRUNCATE tbl1;
TRUNCATE temp_tbl1;

DROP INDEX tbl1_id_idx;

DROP TABLE tbl1;
DROP TABLE tbl2;
DROP TABLE tbl3;
DROP TABLE tbl4;
DROP TABLE temp_tbl1;
DROP TABLE temp_tbl2;
DROP TABLE temp_tbl3;
DROP TABLE temp_tbl4;

SELECT queryid, query FROM pg_stat_statements ORDER BY queryid;

       queryid       |                                 query                                  
---------------------+------------------------------------------------------------------------
                   1 | CREATE TABLE tbl1(id int)
                   2 | CREATE TEMP TABLE temp_tbl1(id int)
                   3 | CREATE TABLE tbl3 AS SELECT x FROM generate_series($1, $2) x
                   4 | CREATE TEMP TABLE temp_tbl3 AS SELECT x FROM generate_series($1, $2) x
                   5 | CREATE INDEX tbl1_id_idx ON tbl1(id)
                   6 | CREATE INDEX temp_tbl1_id_idx ON temp_tbl1(id)
                   7 | VACUUM tbl1
                   8 | TRUNCATE tbl1
                   9 | DROP TABLE tbl1
                  10 | DROP INDEX tbl1_id_idx
                  11 | COPY temp_tbl1 FROM STDIN
                  12 | COPY tbl1 FROM STDIN
                  13 | COPY temp_tbl1 TO STDOUT
                  14 | COPY tbl1 TO STDOUT

F.52.4. Known issues #

F.52.4.1. Single-element array #

If pg_query_id.ignore_constant_arrays is set and array contains single element it automatically converts to binary operator or another form of expression tree node. For example: WHERE a IN (1) during parsing converts to WHERE a = 1.

Currently, we can not track such situations, so for such queries we will have 2 query ids: for single element and for multiple elements. I.e. for WHERE a IN (1) and for WHERE a IN (1, 2), WHERE a IN (1, 2, 3) etc…