Pipelines knowledge bases reference

This reference documentation for Pipelines knowledge bases includes information on the functions and views available in the aidb extension related to knowledge bases.

Views

aidb.knowledge_bases

Also referenceable as aidb.kbs, the aidb.knowledge_bases view provides a list of all knowledge bases in the database. It includes information about the knowledge base name, the model used, and the source data type.

aidb.knowledge_base_stats

Also referenceable as aidb.kbstat, the aidb.knowledge_base_stats view provides current statistics about auto processing for knowledge base pipelines.

Types

aidb.DistanceOperator

The aidb.DistanceOperator type is an enum that represents the distance operators that can be used during retrieval.

SQL definition:

CREATE TYPE DistanceOperator AS ENUM (
  'L2',
  'InnerProduct',
  'Cosine',
  'L1',
  'Hamming',
  'Jaccard'
);

aidb.PipelineDataFormat

The aidb.PipelineDataFormat type is an enum that represents the data formats that can be used as source data.

SQL definition:

CREATE TYPE PipelineDataFormat AS ENUM (
  'Text',
  'Image',
  'Pdf'
);

aidb.PipelineAutoProcessingMode

The aidb.PipelineAutoProcessingMode type is an enum used to define how auto processing for a Pipeline shoud behave (e.g. a knowledge base or a Preparer).

CREATE TYPE PipelineAutoProcessingMode AS ENUM (
	'Live',
	'Background',
	'Disabled'
);

Functions

aidb.create_table_knowledge_base

Creates a knowledge base for a given table.

Parameters

Example

SELECT aidb.create_table_knowledge_base(
               name => 'test_knowledge_base',
               model_name => 'bert',
               source_table => 'test_source_table',
               source_data_column => 'content',
               source_data_type => 'Text'
       );

aidb.create_volume_knowledge_base

Creates a knowledge base for a given PGFS volume.

Parameters

Example

SELECT aidb.create_volume_knowledge_base(
               name => 'demo_vol_knowledge_base',
               model_name => 'simple_model',
               source_volume_name => 'demo_bucket_vol'
       );

aidb.set_auto_knowledge_base

Sets the auto-processing mode for this knowledge base.

Parameters

Example

SELECT aidb.set_auto_knowledge_base('test_knowledge_base', 'Live');
SELECT aidb.set_auto_knowledge_base('test_knowledge_base', 'Disabled', batch_size => 100);
SELECT aidb.set_auto_knowledge_base('test_knowledge_base', 'Background');
SELECT aidb.set_auto_knowledge_base('test_knowledge_base', 'Background', batch_size => 100, background_sync_interval => '10 seconds');

aidb.bulk_embedding

Compute the embeddings for the source records in this pipeline and store them in vector destination.

The behavior of this function depends on the configured auto_processing mode:

• Live and Disabled auto-processing: Directly process all source records.
• Background auto-processing: Mark all source records for processing but don't perform the operation. The background worker will perform it.

Parameters

Example

aidb=# SELECT aidb.bulk_embedding('kb_volume_image_manual');

INFO:  kb_volume_image_manual: (re)setting state table to process all data...
INFO:  kb_volume_image_manual: Starting... Batch size 10, count(known source records): 0, scans completed: 0, count(embeddings): 0
INFO:  kb_volume_image_manual: Batch iteration finished, count(known source records): 10, scans completed: 0, count(embeddings): 10
[...]
INFO:  kb_volume_image_manual: Batch iteration finished, count(known source records): 177, scans completed: 0, count(embeddings): 177
INFO:  kb_volume_image_manual: finished, count(known source records): 177, scans completed: 1, count(embeddings): 177

aidb.retrieve_key

Retrieves a key from matching embeddings without looking up the source data.

Parameters

Example

SELECT * FROM aidb.retrieve_key('test_knowledge_base', 'shoes', 2);

  key  |      distance
-------+--------------------
 43941 | 0.2938963414490189
 19337 | 0.3023805122617119
(2 rows)

aidb.retrieve_text

Retrieves the source text data from matching embeddings by joining the embeddings with the source table.

Parameters

Returns

Example

SELECT * FROM aidb.retrieve_text('test_knowledge_base', 'jacket', 2);

  key  |                       value                        |      distance
-------+----------------------------------------------------+--------------------
 19337 | United Colors of Benetton Men Stripes Black Jacket | 0.2994317672742334
 55018 | Lakme 3 in 1 Orchid  Aqua Shine Lip Color          | 0.3804609668507203
(2 rows)

aidb.delete_knowledge_base

Deletes only the knowledge base's configuration from the database.

Parameters

Example

select aidb.delete_knowledge_base('test_knowledge_base');

 delete_knowledge_base
------------------

(1 row)

aidb.create_volume

Creates a volume from a PGFS storage location for use as a data source in knowledge bases.

Parameters

Example

select aidb.create_volume('demo_bucket_vol', 'demo_bucket', 'demo_bucket/demo_folder', 'Text');

 create_volume
---------------
(1 row)

aidb.list_volumes

Lists all the volumes that were created in the database.

Example

select * from aidb.list_volumes();

aidb.delete_volume

Deletes a volume from the database.

Parameters

Example

select aidb.delete_volume('demo_bucket_vol');

 delete_volume
-------------
(1 row)