Skip to main content

A Comparison of Neural Recommender System Architectures

An independent study of neural recommender architectures, hyperparameter search, and optimization under a common computational budget.

This independent study asks whether increased model and optimization complexity improves recommendation-style edge prediction when architectures, search methods, and optimizers are evaluated under a shared, constrained model-search budget.

Explore the repository ↗ Read the draft manuscript ↗

Three comparisons

  1. Architecture: six model families spanning embedding, graph, sequential, relational-feature, and policy-guided approaches.
  2. Hyperparameter search: grid, random, and Differential Evolution-style search, each with 24 candidate evaluations over the same search domain.
  3. Optimization: Adam and momentum SGD compared with particle swarm and evolution-strategy-style methods.

Making the comparison meaningful

Each method uses the same five trial seeds. Hyperparameters are selected by mean validation AUC, then the selected configuration is retrained and evaluated on the test split. Candidate-search rows contain no test results, keeping the test set out of model selection.

Alongside ROC AUC and average precision, the harness records time and resource use. The aim is to compare predictive performance with the computational cost of obtaining it.

Evaluation datasets

MovieLens 100K supplies the smaller user–item recommendation benchmark. Ratings of 4 or 5 become positive edges; lower ratings are excluded. Positive interactions are ordered by timestamp and split into 80% training, 10% validation, and 10% test data. Negative examples are sampled from unobserved pairs, whose absence leaves user preference unknown.

ogbl-collab, the Open Graph Benchmark author-collaboration dataset, supplies the larger graph-learning proxy using its official link-prediction split. Together, the datasets test the same methods on user–item interactions and a more complex collaboration graph.

Results

The August 25, 2026 run used one NVIDIA RTX PRO 6000 GPU. These are mean test ROC AUC values ± sample standard deviation over five seeds, as reported in the repository.

Comparison MovieLens 100K ogbl-collab
Highest mean AUC, model families DNN: 0.4627 ± 0.0096 PSL-DNN: 0.9477 ± 0.0015
Highest mean AUC, search methods Grid: 0.6562 ± 0.0183 DE-style: 0.9178 ± 0.0051
Highest mean AUC, optimizers PSO: 0.5107 ± 0.0288 Adam: 0.8650 ± 0.0243

The weak MovieLens model-family results are part of the story: the strongest default family in that comparison still scored below 0.5 AUC. Tuning results were better, but the three MovieLens search methods remained close, from 0.6528 to 0.6562.

On ogbl-collab, DE-style search had the highest mean test AUC, while grid search had the lowest measured candidate-search time. These are descriptive comparisons, not formal significance tests.

What I built

A shared PyTorch experiment harness, repeatable dataset preparation and negative sampling, validation-only hyperparameter selection, resource measurements, and separate result files for CPU quick-mode and full-data GPU runs. The repository also includes tests, a notebook that runs the experiments and shows the results, a GPU batch script, and the manuscript.

NoteNotes
  • The manuscript is an unpublished working draft awaiting formal peer review.
  • Hybrid models are compact research approximations of the architectures that inspired them.
  • The full-data run uses short training schedules and sampled, balanced positive-negative mini-batches. Runtime comparisons depend on the hardware and run mode.
  • Evaluation measures sampled edge prediction. Ranking metrics such as NDCG@K and Recall@K remain future work.

Explore further

Back to all projects →