postgresql 在Django模型中使用Trigram(gin_trgm_ops)创建Gin索引

huus2vyu  于 2023-05-17  发布在  PostgreSQL
关注(0)|答案(6)|浏览(247)

django.contrib.postgres的新TrigramSimilarity特性对于我遇到的一个问题来说非常好。我用它作为一个搜索栏来查找很难拼写的拉丁名字。问题是有超过200万个名字,搜索的时间比我想要的要长。
我想在postgres documentation中描述的三元组上创建一个索引。
但我不确定如何以Django API使用它的方式来做到这一点。对于postgres文本搜索,有关于如何创建索引的描述,但没有关于三元组相似性的描述。
这就是我现在所拥有的:

class NCBI_names(models.Model):
    tax_id          =   models.ForeignKey(NCBI_nodes, on_delete=models.CASCADE, default = 0)
    name_txt        =   models.CharField(max_length=255, default = '')
    name_class      =   models.CharField(max_length=32, db_index=True, default = '')

    class Meta:
        indexes = [GinIndex(fields=['name_txt'])]

在视图的get_queryset方法中:

class TaxonSearchListView(ListView):    
    #form_class=TaxonSearchForm
    template_name='collectie/taxon_list.html'
    paginate_by=20
    model=NCBI_names
    context_object_name = 'taxon_list'

    def dispatch(self, request, *args, **kwargs):
        query = request.GET.get('q')
        if query:
            try:
                tax_id = self.model.objects.get(name_txt__iexact=query).tax_id.tax_id
                return redirect('collectie:taxon_detail', tax_id)
            except (self.model.DoesNotExist, self.model.MultipleObjectsReturned) as e:
                return super(TaxonSearchListView, self).dispatch(request, *args, **kwargs)
        else:
            return super(TaxonSearchListView, self).dispatch(request, *args, **kwargs)
    
    def get_queryset(self):
        result = super(TaxonSearchListView, self).get_queryset()
        #
        query = self.request.GET.get('q')
        if query:            
            result = result.exclude(name_txt__icontains = 'sp.')
            result = result.annotate(similarity=TrigramSimilarity('name_txt', query)).filter(similarity__gt=0.3).order_by('-similarity')
        return result
34gzjxbg

34gzjxbg1#

我发现了一个使用最新版本Django ORM的12/2020 article

class Author(models.Model):
    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)

    class Meta:
        indexes = [
            GinIndex(
                name='review_author_ln_gin_idx', 
                fields=['last_name'], 
                opclasses=['gin_trgm_ops'],
            )
        ]

如果像最初的海报一样,你想创建一个与icontains一起工作的索引,你必须索引列的UPPER(),这需要OpClass的特殊处理:

from django.db.models.functions import Upper
from django.contrib.postgres.indexes import GinIndex, OpClass

class Author(models.Model):
        indexes = [
            GinIndex(
                OpClass(Upper('last_name'), name='gin_trgm_ops'),
                name='review_author_ln_gin_idx',
            )
        ]

old article的启发,我使用了current one,它给出了GistIndex的以下解决方案:
更新:从Django-1.11开始,事情似乎更简单了,因为this answer和django文档建议:

from django.contrib.postgres.indexes import GinIndex

class MyModel(models.Model):
    the_field = models.CharField(max_length=512, db_index=True)

    class Meta:
        indexes = [GinIndex(fields=['the_field'])]

从Django-2.2开始,class Index(fields=(), name=None, db_tablespace=None, opclasses=())中将提供一个属性opclasses用于此目的。

from django.contrib.postgres.indexes import GistIndex

class GistIndexTrgrmOps(GistIndex):
    def create_sql(self, model, schema_editor):
        # - this Statement is instantiated by the _create_index_sql()
        #   method of django.db.backends.base.schema.BaseDatabaseSchemaEditor.
        #   using sql_create_index template from
        #   django.db.backends.postgresql.schema.DatabaseSchemaEditor
        # - the template has original value:
        #   "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s)%(extra)s"
        statement = super().create_sql(model, schema_editor)
        # - however, we want to use a GIST index to accelerate trigram
        #   matching, so we want to add the gist_trgm_ops index operator
        #   class
        # - so we replace the template with:
        #   "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s gist_trgrm_ops)%(extra)s"
        statement.template =\
            "CREATE INDEX %(name)s ON %(table)s%(using)s (%(columns)s gist_trgm_ops)%(extra)s"

        return statement

然后你可以在你的模型类中像这样使用它:

class YourModel(models.Model):
    some_field = models.TextField(...)

    class Meta:
        indexes = [
            GistIndexTrgrmOps(fields=['some_field'])
        ]
z6psavjg

z6psavjg2#

我也遇到了类似的问题,试图使用pg_tgrm扩展来支持高效的containsicontains Django字段查找。
可能有一种更优雅的方法,但定义一个新的索引类型对我来说是有效的:

from django.contrib.postgres.indexes import GinIndex

class TrigramIndex(GinIndex):
    def get_sql_create_template_values(self, model, schema_editor, using):
        fields = [model._meta.get_field(field_name) for field_name, order in self.fields_orders]
        tablespace_sql = schema_editor._get_index_tablespace_sql(model, fields)
        quote_name = schema_editor.quote_name
        columns = [
            ('%s %s' % (quote_name(field.column), order)).strip() + ' gin_trgm_ops'
            for field, (field_name, order) in zip(fields, self.fields_orders)
        ]
        return {
            'table': quote_name(model._meta.db_table),
            'name': quote_name(self.name),
            'columns': ', '.join(columns),
            'using': using,
            'extra': tablespace_sql,
        }

方法get_sql_create_template_values是从Index.get_sql_create_template_values()复制而来的,只有一处修改:添加+ ' gin_trgm_ops'
对于您的用例,您可以使用这个TrigramIndex而不是GinIndexname_txt上定义索引。然后运行makemigrations,这将产生一个迁移,生成所需的CREATE INDEX SQL。
更新:
我看到你也在使用icontains进行查询:

result.exclude(name_txt__icontains = 'sp.')

Postgresql后端会把它变成这样:

UPPER("NCBI_names"."name_txt"::text) LIKE UPPER('sp.')

然后由于UPPER(),将不使用三元组索引。
我也遇到了同样的问题,最后我将数据库后端子类化来解决这个问题:

from django.db.backends.postgresql import base, operations

class DatabaseFeatures(base.DatabaseFeatures):
    pass

class DatabaseOperations(operations.DatabaseOperations):
    def lookup_cast(self, lookup_type, internal_type=None):
        lookup = '%s'

        # Cast text lookups to text to allow things like filter(x__contains=4)
        if lookup_type in ('iexact', 'contains', 'icontains', 'startswith',
                           'istartswith', 'endswith', 'iendswith', 'regex', 'iregex'):
            if internal_type in ('IPAddressField', 'GenericIPAddressField'):
                lookup = "HOST(%s)"
            else:
                lookup = "%s::text"

        return lookup

class DatabaseWrapper(base.DatabaseWrapper):
    """
        Override the defaults where needed to allow use of trigram index
    """
    ops_class = DatabaseOperations

    def __init__(self, *args, **kwargs):
        self.operators.update({
            'icontains': 'ILIKE %s',
            'istartswith': 'ILIKE %s',
            'iendswith': 'ILIKE %s',
        })
        self.pattern_ops.update({
            'icontains': "ILIKE '%%' || {} || '%%'",
            'istartswith': "ILIKE {} || '%%'",
            'iendswith': "ILIKE '%%' || {}",
        })
        super(DatabaseWrapper, self).__init__(*args, **kwargs)
isr3a4wc

isr3a4wc3#

这已经有了答案,但在Django 2.2中,你可以更容易地做到这一点:

class MyModel(models.Model):
   name = models.TextField()
   class Meta:
       indexes = [GistIndex(name="gist_trgm_idx", fields=("name",), opclasses=("gist_trgm_ops",))]

您也可以使用GinIndex

sr4lhrrt

sr4lhrrt4#

让Django 2.2使用icontains和类似搜索的索引:
子类GinIndex创建一个不区分大小写的索引(所有字段值都大写):

from django.contrib.postgres.indexes import GinIndex

class UpperGinIndex(GinIndex):

    def create_sql(self, model, schema_editor, using=''):
        statement = super().create_sql(model, schema_editor, using=using)
        quote_name = statement.parts['columns'].quote_name

        def upper_quoted(column):
            return f'UPPER({quote_name(column)})'
        statement.parts['columns'].quote_name = upper_quoted
        return statement

像这样将索引添加到模型中,包括使用opclasses时需要的kwarg name

class MyModel(Model):
    name = TextField(...)

    class Meta:
        indexes = [
            UpperGinIndex(fields=['name'], name='mymodel_name_gintrgm', opclasses=['gin_trgm_ops'])
        ]

生成迁移并编辑生成的文件:

# Generated by Django 2.2.3 on 2019-07-15 10:46
from django.contrib.postgres.operations import TrigramExtension  # <<< add this
from django.db import migrations
import myapp.models

class Migration(migrations.Migration):

    operations = [
        TrigramExtension(),   # <<< add this
        migrations.AddIndex(
            model_name='mymodel',
            index=myapp.models.UpperGinIndex(fields=['name'], name='mymodel_name_gintrgm', opclasses=['gin_trgm_ops']),
        ),
    ]
ef1yzkbh

ef1yzkbh5#

如果有人想在多个列上用空格连接索引,你可以使用我的内置索引。
创建类似gin (("column1" || ' ' || "column2" || ' ' || ...) gin_trgm_ops)的索引

class GinSpaceConcatIndex(GinIndex):

    def get_sql_create_template_values(self, model, schema_editor, using):

        fields = [model._meta.get_field(field_name) for field_name, order in self.fields_orders]
        tablespace_sql = schema_editor._get_index_tablespace_sql(model, fields)
        quote_name = schema_editor.quote_name
        columns = [
            ('%s %s' % (quote_name(field.column), order)).strip()
            for field, (field_name, order) in zip(fields, self.fields_orders)
        ]
        return {
            'table': quote_name(model._meta.db_table),
            'name': quote_name(self.name),
            'columns': "({}) gin_trgm_ops".format(" || ' ' || ".join(columns)),
            'using': using,
            'extra': tablespace_sql,
        }
bnlyeluc

bnlyeluc6#

确保已将django.contrib.postgres添加到INSTALLED_APPS
来源https://code.djangoproject.com/ticket/32770

相关问题