# How to Prefetch Across GenericForeignKeys When You Can't Change the Schema You don't always control the schema you work with. Sometimes you inherit a codebase where `GenericForeignKey` is threaded through the models, and a migration isn't on the table. Loading a page of 20 audit entries runs ~60 queries because Django's ORM can't prefetch across a GFK. Here's how to fix it without changing a single model. --- ## The Problem `GenericForeignKey` lets one table point at rows in multiple other tables without knowing which one at query time. The classic use case is an audit log, a notification feed, or a tagging system where the set of target types is open-ended. But `select_related` and `prefetch_related` can't traverse a GFK. The ORM needs to know the target table ahead of time, and a `content_type_id`/`object_id` pair doesn't give it that until runtime. The result: every time you touch the GFK on an instance, you pay a query. Take an activity log. One table, every row references some target: an order, a user, a product: ```python # models.py from django.contrib.contenttypes.fields import GenericForeignKey, GenericRelation from django.contrib.contenttypes.models import ContentType from django.db import models class ActivityLog(models.Model): actor = models.ForeignKey("User", on_delete=models.CASCADE) action = models.CharField(max_length=50) # "created", "updated", "deleted" content_type = models.ForeignKey(ContentType, on_delete=models.CASCADE) object_id = models.CharField(max_length=255) target = GenericForeignKey("content_type", "object_id") created_at = models.DateTimeField(auto_now_add=True) class Order(models.Model): number = models.CharField(max_length=20) status = models.CharField(max_length=20) logs = GenericRelation(ActivityLog) class Product(models.Model): name = models.CharField(max_length=255) sku = models.CharField(max_length=50) logs = GenericRelation(ActivityLog) class User(models.Model): email = models.EmailField() logs = GenericRelation(ActivityLog) ``` This is a legitimate use of GFK. Audit logs *should* be polymorphic. But rendering a page of 20 activity feed entries, each showing the target's name and a link, hits the database once per entry to resolve `log.target`: 20 individual single-row lookups. If the audit trail also needs actor details, or the target's related data, the count multiplies fast. Twenty entries can easily become 60-80 queries before pagination. --- ## The Solution Three steps. First, batch-resolve every GFK lookup in the page at once using compound queries. Second, attach the results to each instance as a cached private attribute. Third, let serializers read the cache with a live-query fallback, so the optimization is optional and the code never breaks if you forget to run it. ### Step 1: Resolve the targets in bulk, not one by one Collect the log entries you want to display, then group their `object_id` values by content type. Fire one query per target model, map the results back by ID, and attach the resolved instances: ```python from collections import defaultdict from django.contrib.contenttypes.models import ContentType def hydrate_activity_targets(entries): """Resolve all ActivityLog.target references in bulk.""" if not entries: return # Collect all (content_type_id, object_id) pairs from the page refs_by_ct = defaultdict(set) for entry in entries: if entry.content_type_id and entry.object_id: refs_by_ct[entry.content_type_id].add(entry.object_id) # Resolve content types dynamically from the entries. One query, # no hardcoded model list — inherited codebases gain new types # and this adapts automatically. ct_by_id = { ct.id: ct for ct in ContentType.objects.filter(id__in=refs_by_ct.keys()) } # One query per target model, map results back by (ct_id, str(id)) target_map = {} for ct_id, obj_ids in refs_by_ct.items(): ct = ct_by_id.get(ct_id) if ct is None: continue model = ct.model_class() if model is None: continue for obj in model.objects.filter(id__in=obj_ids): target_map[(ct_id, str(obj.id))] = obj # Attach to each entry for entry in entries: key = (entry.content_type_id, str(entry.object_id)) entry._resolved_target = target_map.get(key) ``` This resolves every target reference on the page, regardless of type, with one query per content type. For a feed mixing 4 types, that's 4 queries instead of 20. The `ContentType` map is built from the entries themselves (no hardcoded model list), so the function automatically handles content types added after the hydration code was written. ### Step 2: Handle the related data with compound queries Resolving the target object is only the first layer. Audit entries also need the *actor's* name, or a display string that lives on a related model. You can extend the hydration to fetch those in bulk too. The pattern is the same: bucket by content type, compound `Q` filter across all buckets, map back by `(content_type_id, object_id)`: ```python from django.db.models import Q def hydrate_activity_details(entries): """Bulk-fetch display details for all resolved targets on the page.""" if not entries: return # Build a model → ct_id map up front. Unlike Step 1, display formatting # is inherently type-specific — a product shows name and SKU, a user # shows their full name — so hardcoding known types here is correct. ct_map = ContentType.objects.get_for_models(Product, User, Order) ct_id_by_model = {model: ct.id for model, ct in ct_map.items()} # Group entries by content type obj_ids_by_ct = defaultdict(list) entry_by_target_id = defaultdict(list) # target.id → [entries with that target] for entry in entries: target = getattr(entry, "_resolved_target", None) if target is None: continue ct_id = ct_id_by_model.get(type(target)) if ct_id is not None: obj_ids_by_ct[ct_id].append(target.id) entry_by_target_id[(ct_id, target.id)].append(entry) # Fetch products in one query product_ct = ct_id_by_model.get(Product) if product_ct in obj_ids_by_ct: product_display = { p.id: f"{p.name} ({p.sku})" for p in Product.objects.filter( id__in=obj_ids_by_ct[product_ct] ) } for product_id, display in product_display.items(): for entry in entry_by_target_id[(product_ct, product_id)]: entry._target_display = display # Fetch users with their profile in one query user_ct = ct_id_by_model.get(User) if user_ct in obj_ids_by_ct: user_display = { u.id: u.get_full_name() for u in User.objects.filter( id__in=obj_ids_by_ct[user_ct] ).select_related("profile") } for user_id, display in user_display.items(): for entry in entry_by_target_id[(user_ct, user_id)]: entry._target_display = display # Repeat the same pattern for Order or any other content type. # Targets that don't match a handled type fall back gracefully through # the serializer's str(target) path. ``` Building the display map in a dict comprehension first keeps each pass over entries to O(n). One iteration per content type, not a nested loop over every object × entry combination. For 200 entries and 50 products, that's 50 + 200 checks instead of 50 × 200. ### Step 3: Serializers that read the cache with a fallback Serializers read the cached attributes when they're present, and fall back to a live query when they aren't, so the serializer works whether or not the hydration ran: ```python class ActivityLogSerializer(serializers.Serializer): target_name = serializers.SerializerMethodField() target_type = serializers.SerializerMethodField() def get_target_name(self, entry): display = getattr(entry, "_target_display", None) if display is not None: return display # Cache wasn't populated — fall back to a live GFK lookup. target = entry.target return str(target) if target else None def get_target_type(self, entry): target = getattr(entry, "_resolved_target", entry.target) if target is None: return None return type(target).__name__ ``` The `getattr(entry, "_target_display", None)` pattern is the decoupling mechanism. If hydration ran, the attribute is present and `entry.target` never fires. If it didn't run, the serializer falls back to a live GFK query and everything still works. The hydration logic is an optional optimization — the serializer doesn't care whether it was called. --- ## Why This Works **One query per content type, not per row.** The `id__in` filter resolves all targets of a given type in a single query regardless of how many entries reference that type. A feed with 20 entries across 4 types goes from 20+ queries to 4. **The private-attribute convention is lazy and safe.** `_resolved_target` and `_target_display` are set when hydration runs and silently absent when it doesn't. The `getattr(..., fallback)` pattern means no code path breaks. The serializer degrades to live queries without any conditional branching. **No schema change required.** This pattern doesn't touch the models, doesn't need a migration, and doesn't alter how GFK works. It's a presentation-layer optimization. You can introduce it incrementally to the slowest pages and leave the rest untouched. **It composes with existing `select_related`/`prefetch_related`.** If your queryset already uses ORM-level prefetch for non-GFK relationships (like `select_related("actor")`), the hydration runs *after* that, filling in only the gaps the ORM can't reach. --- ## Design Decision: The Fallback Is Silent, and That's Both a Feature and a Risk Because `getattr(entry, "_resolved_target", entry.target)` swallows a missing attribute silently, a typo in the attribute name (`_resolvedTarget`) will never raise an error. It'll just quietly bypass the cache and fire a live query on every call. The page still renders correctly, but your query count silently regresses. Mitigations: keep the `_` prefix convention consistent, test the query count in integration tests (not just correctness), and consider a debug-mode assertion that logs a warning when hydration is expected but the cache attribute is absent. One more operational note: if the set of target types is large, the `id__in` per-content-type approach fires many small queries rather than one big one. At dozens of content types, a single query that fetches all targets of any type via `UNION` may be faster — but for the typical audit log with half a dozen entity types, the per-type approach is simpler and easier to debug. --- ## The Result An activity feed that used to cost ~60 queries for 20 entries — the initial queryset (1), a per-entry GFK target lookup (20), an actor display-name lookup per entry (20), and a related-field fetch per target (20+, e.g. user profiles, product details) — now costs about 6: one per target content type plus the initial fetch. The two hydration functions compose between the queryset and the serializer: ```python entries = ActivityLog.objects.select_related("actor").order_by("-created_at")[:20] hydrate_activity_targets(entries) hydrate_activity_details(entries) return ActivityLogSerializer(entries, many=True).data ``` No schema change, no migration, no ORM tricks. The hydration functions are drop-ins that run between the queryset and the serializer, and if you forget to call them, nothing breaks. The serializers fall back to live queries. How are you handling N+1 across GenericForeignKeys in your apps? Are you hand-rolling prefetches, or have you found a different pattern? Drop your approach in the comments. ¡Hasta luego!